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I.  INTRODUCTION 


This  report  is  a  synthesis  of  existing  case  study  reports, 
eight  new  case  studies  conducted  specifically  to  document  quanti¬ 
tative  and  qualitative  data  for  cross-program  analysis,  and 
documents,  presentations,  and  other  available  literature.  It 
represents  a  portion  of  a  large  -study  focused  on  readiness  through 
R&M.  It  was  made  possible  only  by  the  extraordinary  support  of 
the  leadership  and  personnel  of  the  Office  of  the  Secretary  of 
Defense,  the  military  services,  government,  industry  d  academia. 

The  overall  IDA  study  is  divided  into  two  raajoi  egments — 
one  to  undertake  case  studies  of  existing  systems,  <  stained  in 
this  volume,  and  Volume  IV  which  examines  existing  ^-tunities 
to  use  new  technology.  This  volume  integrates  the  e  ,nsive  case 
study  efforts  of  a  large-scale  analysis  activity,  provides  signi¬ 
ficant  observations  and  findings  and  identifies  high-payoff  areas 
for  improving  weapon  system  readiness  through  innovative  program 
structuring.  The  relationship  of  this  report  to  the  other  study 
reports  is  indicated  in  Fig.  1. 

The  study  was  done  for  OASD  (MRA&L)  and  OUSD  ( R& E ) . 

Russell  R.  Shorey  (MRA&L)  has  been  the  Department  of  Defense 
point  of  contact  throughout. 

Conclusions,  observations  and  findings  from  the  case  study 
analysis  are  contained  in  Section  IV  of  this  report.  More  detailed 
Information  is  provided  in  the  appendixes  and  individual  case  study 
reports. 
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II.  BACKGROUND 


In  recent  years  there  has  been  a  rising  concern  about  DoD's 
ability  to  keep  weapon  systems  both  modern  and  combat-ready. 

At  any  given  time  the  availability  of  many  of  these  systems  has 
been  below  that  needed  to  maintain  the  required  force  posture. 
The  seriousness  of  this  problem  was  highlighted  in  the  report 
of  the  1981  Defense  Science  Board  (DSB)  study  of  the  Operational 
Readiness  of  High  Performance  Systems.  One  of  the  major  recom¬ 
mendations  of  that  study  was  to  design  reliability  into  the  sys¬ 
tems  from  the  start  and  mature  that  capability  prior  to  full- 
rate  production.  The  1981  DSB  study  also  highlighted  problems 
with  diagnostics  and  recognized  that  increasing  system  complex¬ 
ity,  while  not  incompatible  with  readiness,  made  it  imperative 
that  the  Department  of  Defense  (DoD)  demand  and  manage  acquisi¬ 
tion  to  achieve  readiness  requirements. 

Because  of  the  well  publicized  problems  in  reliability, 
readiness  and  support,  DoD  put  improvements  in  this  area  high 
on  its  priority  list.  The  Carlucci  initiatives  directed  at 
reforming  the  acquisition  process  gave  reliability  and  support 
considerations  a  very  high  priority.  As  a  result  there  has  been 
a  major  increase  in  DSARC  and  top  management  attention.  On  each 
major  program  there  is  visibility  at  the  top  on  progress  in 
meeting  R&M  objectives  through  development,  production  and  in 
early  field  experience. 

The  track  record  from  these  efforts  is  uneven.  Many  of  the 
more  mature  technologies  have  done  relatively  well  in  meeting 
reliability  objectives.  Newer,  fast  developing  technologies 
often  have  serious  problems,  however,  as  do  programs  with 
accelerated  or  compressed  schedules.  The  latter  are  becoming 
more  frequent  due  to  the  Administration  objectives  of  fielding 
new  hardware  faster.  Thus,  there  is  a  major  challenge  in 
learning  to  manage  acquisitions  on  accelerated  programs  so  as 
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to  attain  desirable  R&M  objectives.  Technology  advances  are 
potentially  helpful  in  such  areas  (e.g.,  in  electronics)  by 
providing  opportunities  to  improve  both  performance  and  R&M, 
provided  the  problem  is  attacked  in  both  the  technology  base 
and  the  acquisition  process. 

In  the  future,  increasing  weapon  system  complexity  and 
rising  maintenance  costs  will  lead  to  demands  for  higher  levels 
of  R&M.  A  review  of  the  Services'  Year  2000  studies  identified 
a  common  theme  calling  for  more  flexibility,  more  autonomy, 
more  dispersal,  and  reduced  support  tail  dependency  in  combat 
forces.  While  the  validity  of  the  presumptions  on  which  these 
requirements  are  based  may  be  challenged,  their  general  thrust 
is  unmistakable. 

As  a  result  of  these  concerns,  the  Office  of  the  Assistant 
Secretary  of  Defense  (Manpower,  Reserve  Affairs  and  Logistics) 
and  the  Office  of  the  Under  Secretary  of  Defense  for  Research 
and  Engineering  initiated  this  study,  "Steps  Toward  Improving 
the  Materiel  Readiness  Posture  of  the  DoD"  (short  titles  R&M 
Study)  at  the  Institute  for  Defense  Analyses  (IDA)  with  the 
purpose  of  identifying  and  providing  support  for  high-payoff 
actions  which  the  DoD  can  take  to  improve  system  design,  devel¬ 
opment  and  support  processes  so  as  to  provide  quantum  improve¬ 
ments  in  R&M  and  readiness  through  innovative  uses  of  advancing 
technology  and  program  structure  (Appendix  A,  Task  Order). 
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III.  OBJECTIVE  AND  SCOPE 


The  case  study  portion  of  the  overall  study  addressed  two 
distinct  tasks: 

(1) .  To  assess  the  impact  of  program  structuring 

on  future  DoD  requirements  for  improved  R&M 
readiness;  and 

(2) .  To  evaluate  the  potential  and  recommend  strat¬ 

egies  that  might  result  in  quantum  increases 
in  R&M  readiness  through  innovative  use  of 
program  structuring. 

Eight  specific  programs  were  selected  for  study  as  follows 


Program 


APG-63  Radar 
APG-65  Radar 
APG-66  Radar 
T700  Engine 
ASN-128  Radar 
TPQ-36  Radar 
TPQ-37  Radar 
SPY-l-A  Radar 


Document  No. 


D-19 


D-20 
D-2 1 
D-22 
D-2  3 
D-2  4 
D-2  4 


In  addition,  many  other  programs  and  associated  reports  were  re 
viewed  for  specific  information. 


*See  Appendix  A  Reference  List 
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To  provide  a  framework  against  which  other  programs  can  be 
compared  in  terms  of  applicability  of  observations,  conclusions, 
and  findings  the  following  points  are  provided, 

(1)  Case  Study  Development  Programs.  The  systems  examined 
by  the  case  studies  were  second  and  third  generation 
radars  and/or  engines  as  opposed  to  a  first  generation 
system  like  electrooptical .  The  government  and  con¬ 
tractors  all  had  experienced  personnel  and  relatively 
good  data  bases  to  draw  on  for  the  then  current  genera¬ 
tion  of  equipment. 

(2)  Concurrency.  Varying  degrees  of  concurrency  (overlap 
between  development  and  production  program  phases)  were 
examined.  Observed  ranges  went  from  little  concurrency 
on  the  T700  engine  program  to  a  high  degree  of  concur¬ 
rency  on  the  APG-63  and  APG-66  programs. 

(3)  Disciplined  Approach.  A  disciplined  approach  was  de¬ 
veloped,  documented,  and  used  to  analyze  the  selected 
programs.  (Ref.  Appendix  B  and  IDA  Record  Document 
D-26 .  ) 

A  detailed  description  of  the  study  organization,  partici¬ 
pants,  methodology  and  program  selection  process  is  contained  in 
Appendix  A.  Appendix  B  documents  a  disciplined  approach  to 
planning  and  analyzing  weapon  system  programs  developed  and  used 
during  the  course  of  this  study.  The  Electronic  and  Mechanical 
Weapon  Systems  Programs  were  reviewed,  analyzed  and  the  results 
detailed  in  Appendixes  C  and  D,  respectively.  Appendix  E  expands 
on  the  diagnostics  conclusions  in  this  report  and  examines  diag¬ 
nostics  requirements.  Other  subject  areas  such  as  affordability 
and  test  assets  issues  were  analyzed  and  those  results  are  in 
Appendix  F.  Information  System  Observations  are  contained  in 
Appendix  G  and  finally,  Appendix  H  is  a  collection  of  the  official 
data  used  to  analyze  the  various  parts  of  program  structure  during 
the  course  of  the  study. 
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IV.  CONCLUSIONS,  OBSERVATIONS  AND  FINDINGS 


Numerous  findings  and  observations  resulted  from  the  case 
studies.  This  section  will  highlight  the  most  significant. 

Major  conclusions  from  the  case  study  analysis  are: 

•  There  are  high-payoff  actions  currently  known  that 
must  be  planned  for  and  retained  in  the  weapon  system 
acquisition  process  and  programs.  These  include: 

•  Changed  management  practice  to  reflect  inter¬ 
dependencies  in  acquisition  programs 

•  Greater  use  of  reliability  design  tools  and 
processes  particularly  computer-aided  design 
(CAD)  and  environmental  stress  screening  (ESS) 

•  Establishing  "Off-Line"  maturing  of  key  techno¬ 
logy  including  comprehensive  R&M  growth  and 
maturation  programs  in  the  acquisition  process 
and 

•  Increasing  fundamental  work  in  all  areas  of 
diagnostics  development 

•  Accurate  and  detailed  engineering  quality  information 
on  system  and  component  failures  must  be  provided  for 
identifying  and  solving  equipment  problems  and  focusing 
technology  efforts. 

•  Actions  to  enhance  and  expand  R&M  knowledge  and  experi¬ 
ence  of  DoD  and  industry  engineers  and  managers  must 

be  taken  to  achieve  long-term  improvement  for  the 
full  range  of  weapon  system  acquisitions. 
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A.  HIGH-PAYOFF  ACTIONS 


The  individual  observations  and  findings  for  each  of  these 
areas  are  presented  as  follows: 

1 .  Structure  to  Manage  Interdependent  Program  Elements 

a.  Observations :  The  R&M  elements  of  the  acquisition  pro¬ 
cess  are  well-known;  however,  the  interrelationships  and  depen¬ 
dencies  of  elements  and  subelements  are  less  well  understood. 

As  a  consequence,  management  decisions  have  traded  away  R&M  program 
elements  for  dollars  and/or  schedule  savings  which  ultimately  lead 
to  costly  overruns,  schedule  delays  and  downstream  logistics 
prob lems . 

Because  of  the  much  publicized  problems  in  reliability, 
readiness  and  support,  DoD  put  improvements  in  this  area  high  on 
its  priority  list.  The  Carlucci  initiatives  (Fig.  2)  directed  at 
reforming  the  acquisition  process  gave  reliability  and  support 
considerations  a  very  high  priority.  As  a  result  there  has  been 
a  major  increase  in  DSARC  and  top  management  attention.  On  each 
major  program  there  is  visibility  at  the  top  on  progress  in  meeting 
R&M  objectives  through  development,  production,  and  in  early 
field  experience. 

The  track  record  from  these  efforts  is  uneven.  Many  of  the 
more  mature  technologies  have  done  relatively  well  in  meeting 
reliability  objectives.  Newer,  fast  developing  technologies 
often  have  serious  problems,  however,  as  do  programs  with 
accelerated  or  compressed  schedules.  The  latter  are  becoming 
more  frequent  due  to  the  Administration  objectives  of  fielding 
new  hardware  faster.  Thus  there  is  a  major  challenge  in  leader¬ 
ship  to  manage  acquisitions  on  accelerated  programs  so  as  to 
attain  desirable  R&M  objectives.  Technology  advances  are 
potentially  helpful  in  such  areas  (e.g.,  in  electronics)  by 
providing  opportunities  to  improve  both  performance  and  R&M 
provided  the  problem  is  attacked  in  both  the  technology  base 
and  the  acquisition  process. 
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A  simplified  view  of  the  overall  program  process  is  shown 
in  Fig.  3. 

Key  elements  to  this  overall  process  are: 

•  Defining  what  is  really  needed. 

e  Providing  sufficient  and  timely  resources. 

e  Utilizing  good  design,  manufacturing  and 
testing  practices. 

•  Providing  timely  and  accurate  feedback  in 
the  total  process  and  taking  action  to 
identify  problems  and  effect  solution. 

e  Providing  rapid  and/or  flexible  change 
processing  to  permit  problem  resolution. 

b.  Findings:  The  management  challenge  posed  by  the  inter¬ 
dependency  issue  is  to  structure  a  single  acceptable  disciplined 
approach  to  planning  programs  to  assure  balanced  considerations 
of  performance,  budget,  schedule  and  supportability .  Once  programs 
are  so  structured,  the  discipline  must  provide  for  continuing 
analysis  and  monitoring  to  assure  that  the  appropriate  balance 
is  not  lost  as  the  program  progresses  through  its  various  phases. 

This  disciplined  approach  must  also  recognize  the  dependency 
relationship  among  the  program  elements.  For  example,  good 
reliability  predictions  depend  on  a  good  definition  of  how  the  hard¬ 
ware  will  be  used  and  subsequently,  a  good  environmental  analysis. 
These  dependencies  result  in  many  elements  being  "necessary"  but 
few  (or  none)  being  "sufficient,"  in  and  of  themselves,  to  achieve 
satisfactory  or  ultimate  performance.  The  structured  process 
may  have  more  than  one  path  to  success  but  numerous  paths  exist 
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that  will  lead  to  problems  that  will  result  in  unsatisfactory 
reliability  or  maintainability. 


c.  Action :  The  Services  should  analyze  and  develop  a 
discipline  of  managing  interdependent  program  elements,  including 
appropriate  data  bases  and  parameters,  and  include  these  in  their 
acquisition  strategy. 

2.  Reliability  Design  Tools  and  Processes 

a.  Observations :  Design  actions  must  identify  and  balance 
the  stresses  on  various  elements  of  the  equipment.  Reliability 
design  tasks  include  environmental  estimation,  stress  analyses, 
part  selection  and  part  derating.  These  tasks,  in  combination, 
define  (or  estimate)  the  operating  environment  of  the  equipment, 
predict  the  stress  on  the  individual  part  or  component,  select 
a  part  that  can  operate  effectively  in  that  environment  and,  in 
the  case  of  most  electronics  items,  derate  the  part  to  provide  a 
margin  of  safety  between  the  rated  stress  and  the  estimated 
operating  stress.  These  activities  are  fundamental  to  producing 
a  reliable  design.  CAD  has  the  potential  to  make  R&M  a  part  of 
the  mainstream  design  engineering  by  including  R&M  as  a  design 
requirement  and  having  integrated  R&M  design  capabilities. 

Even  if  R&M  design  procedures  are  improved  it  must  be  ex¬ 
pected  that  most  types  of  manufactured  items  will  initially  have 
some  part  and  workmanship  defects.  To  prove  out  manufacturing 
processes  before  fielding,  environmental  stress  screening  (ESS) 
is  needed.  Even  if  R&M  design  procedures  are  improved  it  must 
be  expected  that  most  types  of  manufactured  items  will  initially 
have  some  part  and  workmanship  defects.  The  ESS  approach  is  to 
apply  thermal,  electrical  and  mechanical  stress  to  precipitate 
failure  of  the  weak  parts  and  assemblies  in  the  factory  and 
thereby  result  in  improved  reliability  in  field  use.  All  the 
electronics  programs  in  the  case  studies  used  ESS  to  some  degree 
(see  Fig.  4).  There  was  considerable  variation  in  the  details 
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of  the  applicat ions ,  hut  all  programs  benefited  through  improved 
reliability.  Other  studies  have  shown  that  ESS  reduces  manufac¬ 
turing  costs  and  significantly  improves  productivity  because  of 
reduction  in  rework  and  associated  retesting.  Dramatic  increases 
in  operational  reliability  due  to  ESS,  some  more  than  10  to  1, 
were  also  documented  (see  Fig.  5). 

b.  Findings ;  The  findings  for  ESS  and  CAD  are  as  follows. 
The  design  system  needed  is  computer-aided  design  (CAD)  supported 
by  an  R&M  data  base  and  tied  to  computerized  R&M  and  logistics 
analyses.  Integration  of  R&M  tools  and  analyses  into  CAD  will 
provide  design  engineers  the  disciplined  use  of  specialized 
knowledge  in  real  time  with  potentially  dramatic  reductions  in 
cost. 

In  much  the  same  way  the  integration  of  computer-aided  manu¬ 
facturing  (CAM)  with  CAD  can  enhance  both  design  and  manufactur¬ 
ing.  Integrated  CAD/CAM  can  provide  designers  with  knowledge 
of  manufacturing  constraints  which  can  lead  to  a  more  consistent 
production  process. 

Development  of  an  ESS  approach  should  begin  during  the 
design  phase  and  continue  during  the  development  phase.  ESS 
should  also  be  used  on  test  hardware  so  that  expensive  tests  are 
not  delayed  due  to  design  and  workmanship  problems.  Results 
should  be  analyzed  to  provide  information  needed  to  refine 
screens  prior  to  beginning  production.  The  ability  to  adjust, 
add  or  delete  screens  is  necessary  to  achieve  the  ultimate  ESS 
benefits . 

ESS  applied  during  early  production  serves  as  a  "find-and- 
fix"  program  in  which  manufacturing  process  problems  and  some 
latent  design  deficiencies  are  identified  and  corrected.  Stress 
screens  should  be  determined  after  consideration  of  the  process 
controls  which  can  prevent  introduction  of  manufacturing  de¬ 
fects  and  after  evaluation  of  test  and  inspection  approaches. 
Stress  screens  should  not  be  used  in  lieu  of  possible  preven¬ 
tative  action,  since  preventative  action  is  almost  always  less 
expensive  and  usually  results  in  a  more  reliable  product. 
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The  reliability  potential  of  ESS  is  so  significant  that  it 
warrants  special  attention.  All  electronics  development  and 
production  programs  should  reguire  ESS.  ESS  applications  should 
be  described  in  a  plan  and  must  be  dynamic  in  nature  and  struc¬ 
tured  so  that  maximum  screening  effectiveness  is  obtained. 

Cost  models,  yield  and  rework  data  and  failure  data  should  be 
maintained  to  demonstrate  effectiveness  of  the  program.  Periodic 
reports  showing  the  status  of  screening  results  should  be  provided 
to  management. 

c.  Action;  DoD  should  invest  in  CAD  systems  and  in  ESS 
approaches  that  address  R&M  problems  in  order  to  mature  and 
understand  their  use.  Demonstration  programs  should  be  selec¬ 
tively  funded  and  carefully  evaluated.  Also,  a  policy  should 
be  formulated  to  ensure  ESS  is  applied  to  all  acgui s it  ions . 

3.  "Off-Line"  Maturina  of  Subsystems  and  Components 


a.  Observations ;  Within  a  total  weapon  system  program 
context,  the  off-line  maturing  of  system  and  subsystem  elements 
offers  significant  risk  reduction  for  today's  concurrent  program 
environment.  The  T700  engine  program  represents  a  classic  ex¬ 
ample  of  successful  off-line  development  and  insertion  into  a 
program  with  provisions  for  maturation  (see  Fig.  6). 


b.  Findings :  In  addition  to  this  observation  from  the 
T700  program,  the  technology  portion  of  this  study  recommended 
three  essential  features  of  "off-line"  maturing  that  are  re¬ 
emphasized  here.  The  first  is  that  a  set  of  technologies  should 
be  matured  in  a  manner  which  reflects  their  interdependencies. 
Second,  the  target  chosen  to  provide  the  measure  of  success 
should  be  as  realistic  as  possible,  if  not  improvement  to  an 
existing  system.  Third,  the  results  achieved  should  be  gener¬ 
alized  and  become  the  new  level  of  acceptable  performance. 
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From  a  programmatic  point  of  view,  the  management  issue  becomes 
one  of  when  is  the  technology  ready  for  program  insertion  and 
what  actions  will  minimize  the  risk  of  doing  it.  There  is  no 
simple  answer  for  these  questions  but  the  integrated  approach 
to  technology  maturation  discussed  in  Volume  IV,  Section  V, 
coupled  with  evolution  of  the  disciplined  structured  approach 
as  discussed  in  Volume  III,  Appendix  R,  could  result  in  a 
significant  reduction  in  risk  for  new  programs. 

c.  Action ;  The  need  for  off-line  component  and  subsystem 
development  should  be  evaluated  on  each  DSARC  program  as  well 
as  on  less  major  systems.  Guidelines  should  be  developed  for 
concurrent  programs  to  routinely  fund  such  developments. 

The  Services  should  also  increase  their  technology  base 
efforts  for  programs  with  objectives  such  as  the  Air  Force 
ultra-reliable  radar  program. 

4.  R&M  Growth  and  Maturation 


a.  Observations :  Without  exception,  the  case  studies 
showed  that  despite  the  best  design  efforts,  problems  will  be 
found  in  development  testing,  production,  and  in  field  use.  An 
example  from  the  APG-66  program  is  typical  (Fig.  7).  These 
facts  support  the  rationale  that  testing  and  growth  programs 
are  essential  elements  to  producing  reliable  equipment. 

R&M  growth  programs  should  be  oriented  to  supplement  effective 
design  and  manufacturing  processes. 

A  comprehensive  growth  and  maturation  program  is  more  than 
just  a  test  phase  labeled  reliability  growth  or  reliability  de¬ 
velopment  testing.  It  is  a  coordinated  effort  starting  in  the 
conceptual  phase,  influencing  the  design  phase,  reaching  across 
the  whole  test  program  and  extending  a  reasonable  period  of  time 
into  field  use.  A  well-executed  growth  and  maturation  program 
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requires  adequate  resources  for  data  gathering,  data  review  and 
analysis,  and  for  engineering  manpower  assigned  to  investigate, 
resolve  and  correct  problems  as  well  as  an  expedited  change 
processing  system  to  allow  rapid  incorporation  of  problem 
correct  ions . 

It  is  essential  that  built-in  test  (BIT)  development  be 
structured  to  have  a  parallel  growth  and  maturation  program 
(see  p.  IV-21  for  diagnostics  observations).  The  BIT  in  all  of 
the  cases  studied  required  a  period  of  concentrated  maturation 
before  the  BIT  performance  reached  an  acceptable  level. 

It  is  clear  from  some  programs  reviewed  that  additional 
classes  of  problems  occur  when  equipment  enters  production  and 
again  when  it  enters  operational  service  (Fig.  8).  These 
problems  must  be  identified  and  corrected,  or  the  system  will 
be  plagued  with  the  problems  for  its  entire  operational  life. 

The  continued  growth  and  maturation  of  the  equipment  requires 
technical  personnel  with  understanding  of  the  design.  The 
assignment  of  contractor  personnel  to  the  early  operational 
sites  can  have  a  significant  payoff  in  the  continued  growth  and 
maturation  process. 

b.  Findings :  A  comprehensive  R&M  growth/maturation  pro¬ 
gram  starts  early  in  the  conceptual  planning  phase,  continues 
through  the  design  phase,  influences  component  development 
testing  and  continues  into  the  operational  phase.  The  programs 
evaluated  in  these  case  studies  showed  only  a  limited  amount 
of  growth  planning  and  testing.  The  reliability  improvement 
warranty  (RIW)  programs  on  the  APG-66  and  the  LDNS  were  the 
only  planned  efforts  that  extended  for  any  significant  time 
into  the  operational  phase.  The  following  sections  will  provide 
perspective  on  the  front-end  design  effort  and  the  plans 
necessary  to  manage  a  growth  program  adequately. 
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Appendix  B  of  this  volume  identifies  the  structure  for 
designing  R&M  into  weapons  system  programs.  The  R&M  elements 
identified  were  considered,  to  various  degrees,  in  all  the  case 
studies  evaluated.  It  is  generally  concluded  that  both  the 
military  and  industry  know  how  to  apply  these  elements  to  achieve 
a  high  level  of  "designed  in"  R&M  although  this  effort  by  itself 
will  not  yield  the  R&M  potential  of  the  system  (see  Fig.  9  for 
summary  of  design  analysis). 

System  complexity  generates  problems  which  cannot  be  identi¬ 
fied  by  analysis  only.  These  complexity  problems  arise  from  (1) 
variations  in  the  design  environments  from  actual  operational 
environments,  and  (2)  constraints  on  resources  to  accomplish  an 
accurate  detailed  analysis. 
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The  integration  of  the  total  system  gives  rise  to  interactions 
between  subsystems  that  are  not  identifiable  prior  to  system  integra 
tion.  These  interactions  include  the  interplay  with  other  hardware 
systems,  and  software.  Software  itself  can  create  problems  that 
are  difficult  to  identify.  Diagnostics  falls  into  this  category 
and  a  considerable  testing  effort  is  needed  to  perfect  a  system  to 
the  point  that  it  demonstrates  a  minimum  number  of  false  indica¬ 
tions  of  failure.  Results  from  the  case  studies  indicate  that 
the  front-end  design  effort  led  to  higher  levels  of  R&M  at  the 
start  of  the  test  phase. 

( 2 )  Management  Environment  for  R&M  Growth 

DoD  policy  should  and  contracts  should  provide  for 
integrated  R&M  growth  programs  with  proper  incentives  for  R&M 
growth.  For  major  acquisitions,  DoD  policy  should  include 
planned  operational  R&M  maturation  including  provision  for 
on-site  contractor  design  engineers,  to  investigate  R&M  problems 
and  to  provide  the  detailed  information  necessary  to  develop 
solutions.  Acquisition  programs  must  improve  their  responsiveness 
to  engineering  change  procedures  (see  Fig.  10  for  variances  in 
flow  duration).  Imposition  of  customer  control  of  the  change 
process  should  be  delayed  as  long  as  is  practicable.  Guidelines 
should  be  developed  for  tailoring  management  of  ECPs  during 
the  R&M  change  introduction  process.  Guidelines  should  be  taught 
to  managers  and  enforced  at  all  levels.  Programming  and  budgeting 
should  include  resources  for  data  collection,  analysis,  design 
of  fixes,  and  verification  during  the  production  phase. 

To  achieve  the  R&M  potential  of  a  system,  a  structured 
growth  program  must  be  planned  to  begin  early  in  the  program. 

This  planned  program  must  include  the  identification  of  the  re¬ 
liability  target,  the  number  of  test  assets  and  phases  considered 
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necessary  to  reach  this  potential  target  and  consideration  of 
the  hardware  generation  to  be  used  in  each  phase.  Up-front 
planning  is  necessary  to  assure  that  adequate  resources  are  made 
available  and  that  program  management  understands  the  process 
to  be  followed.  The  general  procedures  in  managing  a  growth 
program  are  defined  in  MIL-HDBK  189,  "Reliability  Growth 
Management."  A  similar  requirement  for  structured  approach  exists 
for  diagnostics  which  has  a  combined  hardware  and  software  system 
growth  requirement. 
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The  growth  must  start  with  the  beginning  of  hardware 
testing  and  extend  two  to  three  years  into  field  operation.  A 
typical  growth  profile  is  shown  on  Fig.  11.  A  test-analyze-and- 
fix  philosophy  must  prevail  throughout  the  program.  Each  phase 
must  make  efficient  use  of  the  test  resources.  After  each  phase 
the  impact  of  corrective  actions  is  evaluated  and  a  management 
decision  made  regarding  the  adequacy  of  the  test  phase  with 
repeat  testing  on  corrected  hardware  as  a  possible  outcome. 

An  accurate  and  timely  failure  feedback  procedure  must 
be  established  to  provide  the  designer  the  necessary  information 
to  accomplish  the  required  corrective  action.  During  early  field 
operation  this  will  require  on-site  engineering  support.  This 
on-site  support  is  necessary  to  ensure  accurate  information  of 
the  failure  causes  to  support  redesign. 

A  prime  requirement  to  ensure  timely  corrective  action  in 
the  field  operation  phase  is  a  rapid  ECP  process.  In  today's  en¬ 
vironment  it  sometimes  takes  as  long  as  two  years  or  more  before  a 
corrective  action  will  get  into  the  hardware.  Methods  for  speeding 
this  process  must  be  an  integral  part  of  the  growth/maturation 
phase . 

Once  the  potential  levels  have  been  achieved  the  system 
must  be  monitored  to  prevent  them  from  degrading  during  production 
and/or  changing  field  environments. 


( 3)  Maturing  the  Support 


Today's  complex,  h ighly-interact ive  systems  require 
the  growth  and  refinement  of  other  factors  to  realize  the  full 
potential  of  reliability  and  maintainability.  These  factors 
include:  diagnostics,  manpower  and  human  factors,  training. 
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technical  data,  support  equipment,  and  software.  The  complex 
interactions  of  man-machines-systems  result  in  the  readiness 
and  sustainability  of  military  systems.  Very  significant  differ¬ 
ences  in  readiness  and  sustainability  can  be  derived  if  the 
total  support  structure  is  matured  in  a  systematic  way.  Growth 
and  maturation  programs  must  be  thought  of  in  this  broader  context. 


( 4 )  Funding 


The  growth  and  maturation  program  for  reliability  and 
diagnostics  is  critically  dependent  on  front-end  funding,  which 
must  be  structured  with  full  identification  of  the  activities 
discussed  above  and  with  particular  attention  given  to  adjunct 
test  hardware. 

A  conceptual  view  of  current  versus  needed  funding 
profiles  is  given  in  Fig.  12.  Further  analysis  would  be  needed 
to  define  the  details  of  an  actual  funding  profile;  Figure  12  is 
intended  only  to  visualize  an  apparent  current  problem. 


funding 

RATE 


FIGURE  12.  Funding  Profile  for  RDT&E 
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c.  Action ;  R&M  Program  MIL-Standa rds  should  be  revised  to 
include  a  plan  for  an  integrated  development  and  field  R&M  growth 
plan. 

5.  Diagnostics 

a.  Observations ;  Diagnostic  systems  development  is  an 
immature  discipline  when  compared  to  reliability.  In  diagnos¬ 
tics,  there  are  no  accepted  definitions  of  requirements  that 
can  be  used  for  contracting  that  are  directly  understandable 
to  a  designer  and  that  can  be  related  to  field  performance. 

On  the  other  hand,  for  reliability,  there  are  design  tools  for 
analysis  of  the  stresses  that  cause  failures  as  well  as  for 
predicting  failure  rates  of  components,  subsystems  and  systems. 
Design  tools  for  diagnostics  are  much  less  structured  and 
pract iced. 

In  reliability  testing,  there  are  proven  techniques  for 
simulating  the  operational  stresses  an  equipment  will  undergo, 
weeding  out  the  causes  of  unreliability  and  verifying  the  poten¬ 
tial  system  reliability.  Diagnostics  testing  techniques  are 
much  less  mature.  Though  fault  insertion  tests  are  performed 
in  the  lab,  they  are  poor  predictors  of  field  performance.  A 
comparison  of  results  from  laboratory  fault  insertion  tests 
and  field  operational  tests  is  shown  in  Fig.  13.  It  will  iden¬ 
tify  some  problems,  but  success  in  such  a  demonstration  is  no 
guarantee  of  a  good  design.  Thus,  demonstration  by  fault 
insertions  are  necessary,  but  not  sufficient,  to  validate  a 
diagnostics  design. 

During  the  early  operational  life  of  a  system  the  assess¬ 
ment  of  reliability  performance  is  much  more  straightforward 
than  diagnostics  performance.  While  there  are  problems  using 
field  data  to  assess  reliability,  such  data  does  provide  some 
management  information  and  can  be  used  for  trend  assessments. 
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Such  maintenance  data  systems  are,  however,  not  useful  for 
diagnostics  evaluation  purposes,  since  the  data  does  not  reflect 
the  method  of  fault  detection/isolation,  human  intervention  in 
the  decision  process,  or  troubleshooting  time  factors. 

Lack  of  knowledge  in  the  diagnostics  area  (contracting, 
statement  of  requirements,  design,  testing,  deployment)  presents 
a  significant  challenge  to  the  development  community  to  improve 
diagnostics  of  current  weapon  systems  and  acquisition  methods 
for  improved  diagnostics  in  future  weapon  systems.  Fundamental 
work  is  required  in  all  these  facets  of  weapon  program  develop¬ 
ment  to  produce  acceptable  diagnostic  capabilities  for  field  use. 

b.  Findings:  From  the  case  study  activities  and  numerous 
other  studies  and  presentations  reviewed,  it  is  clear  that  the 
achievement  of  a  mature  diagnostics  capability  is  the  result  of 
a  defined  process.  This  process  encompasses  both  research  and 
development  activities,  which  are  not  weapon  program  specific, 
as  well  as  the  acquisition  process,  which  can  be  weapon  program 
specific.  Achieving  effective  diagnostics  requires  a  plan,  man¬ 
agement  strategy,  motivation,  technical  activity,  and  funding 
that  spans  system  acquisition  from  initial  requirements  defini¬ 
tion  through  deployment. 

( 1 )  Statement  of  Requirements 

The  military  user's  requirements  should  address  diagnostic 
capability  in  the  larger  context  of  the  operational  mission  and 
environment  as  well  as  the  support  constraints  of  manpower,  the 
skill-level  maintenance  concept,  deployment,  and  the  logistics 
burden.  The  requirements,  constraints,  environment,  and  econom¬ 
ics  should  then  drive  the  architecture  of  the  system,  diagnos¬ 
tics  being  one  of  the  fundamental  characteristics.  Significant 
information  improvements  are  needed  for  formulating  these 
requirements. 
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There  are  a  number  of  innovative  Service  efforts  to  define 
requirements  and  development  objectives  for  readiness  and  support 
at  the  front  end  of  a  weapon  program.  Examples  include  the  Air 
Force  Advanced  Tactical  Fighter  ( ATF ) ,  the  Navy  Submarine  Advanced 
Combat  System  (SUBACS),  the  Army  Mobile  Protected  Gun  System 
(MPGS),  and  the  Joint  Services  Advanced  Vertical-Lift  Aircraft 
(JVX).  Figure  14  summarizes  the  kind  of  performance-driven 
support  requirements  and  constraints  which  define  the  context  for 
generating  diagnostics  requirements.  Objectives  which  call  for 
reduced  levels  of  maintenance,  high  utilization  rates,  self- 
sustaining  operations  and  reduced  support  tail  should  drive  the 
development  of  high-confidence  built-in-test  with  CND/BCS/RTOK 
rates  of  near  zero. 


Non-weapon  Specific 


Military  Needs 


R&D  Activities 


-  Defense  Laboratories 


-  Contractor  IR&D 


-  Technology  Opportunities 


-  Threat 


-  Force  Structure 


-  Needs 


Weapon  System  Program 

-  Requirements 

-  Alternative  Concepts 

-  Demonstration/Validation 

-  Full  Scale  Development 

-  Production/Deployment 
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Definitions,  terminology,  and  figures  of  merit  to  describe 
diagnostics  requirements  have  proliferated  to  the  point  that 
communication  relative  to  diagnostics  measures  is  difficult. 

This  is  not  a  trivial  problem;  it  impedes  the  way  diagnostics 
are  specified,  managed,  designed,  tested,  and  measured.  Proposed 
MIL-STD-XXX,  "Testability  Program  for  Electronic  Systems  and 
Equipment,"  and  MIL-STD-1309  are  useful  but  not  sufficient  steps 
to  resolve  this  problem.  Better  ways  of  specifying  diagnostics 
requirements  are  needed  to  achieve  the  readiness  and  support 
goals  of  the  Services.  One  proposal  was  reviewed  which  appears 
to  have  the  capability  of  influencing  reliability,  maintainability, 
support  costs,  and  readiness  in  a  manner  to  achieve  a  two-level 
maintenance  capability.  The  approach  specifies  that  all  avionics 
line-replaceable  units  meet  a  specific  threshold  of  acquisition 
cost  per  removal-free  operating  hour.  This  parameter  has  the 
advantage  of  being  operationally  useful  and  measurable  in  the 
field  though  it  may  not  communicate  requirements  clearly  to  the 
designer. 


( 2)  Design 

In  the  area  of  diagnostics  system  design,  the  following 
needs  have  been  identified: 

a.  Strategies  to  minimize  cannot  duplicate  (CND),  bench- 
checked  serviceable  (BCS),  retest  OK  (RTOK),  and  false 
alarm  conditions  during  design. 

b.  Techniques  to  maximize  vertical  testability. 

c.  A  flexible  diagnostic  system  so  that  changes  can  be 
incorporated  readily  in  diagnostic  algorithms,  screens, 
and  tolerances  with  minimal  hardware  impact. 
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Fault-free  software  development  techniques. 

Techniques  to  enable  more  concurrent  hardware  and 
software  development  and  earlier  integration  of  the 


f.  Trade-off  tools  for  assessing  the  diagnostics  impli¬ 
cations  of  design  decisions  on  the  support  structure. 

g.  Computer-aided  engineering  techniques  for  enhancing 
design  for  testability  in  support  of  proposed  MIL- 
STD-XXX.  (Some  techniques  such  as  LOGMOD  and  STAMP 
may  already  be  able  to  meet  this  need,  though  they 
are  not  widely  used.) 

h.  Both  the  Services  and  contractors  need  to  develop 
experienced  people  who  understand  how  to  achieve  good 
diagnostics  designs. 

i.  Tools  for  predicting,  measuring,  and  managing  the 
diagnostics  designs. 

j.  Better  design  practices  such  as  control  of  timing 
margins  in  high-speed  circuits  and  systems. 

(3)  Development  and  Demonstration  Test 


Improvements  in  development  and  demonstration  testing  will 
aid  diagnostics  development.  The  following  measures  have  been 
suggested  by  experts  in  the  field: 

a.  Use  reliability  and  other  test  events  as  opportun¬ 
ities  to  discover  problems  with  BIT  performance. 


822/1-26 


IV-26 


TTT7TT 


1  —  1  —  ?  -  7 


■/  ■ 


Environmental  testing  may  be  particularly  useful 
for  discovering  false-alarm  indications  such  as 
induced  intermittents  and  transients. 

b.  Increase  the  number  of  spare  assets  and  the  time 
budgeted  in  the  system  integration  laboratory  to 
investigate  diagnostic  anomalies  without  impacting 
the  schedule  and  use  of  other  assets. 

c.  Expand  the  set  of  faults  inserted.  (Time  required 
for  fault  insertion  tests  might  have  to  increase.) 


d. 


e. 


Increase  the  allowable  cost  of  demonstrations  to 
include  repair  costs.  This  action  will  permit  the 
insertion  of  a  better  cross-section  of  faults. 
Develop  a  library  of  computer  simulation  models  to 
test  BIT  (hardware,  software,  firmware). 


f.  Adopt  comparability  analysis  as  a  useful  tool  for 
identifying  a  realistic  fault  set  for  insertion. 

g.  Develop  and  incorporate  in  MIL-STD-471  improved 
demonstration  techniques  to  predict  field  diagnostics 
performance . 

( 4)  Operational  Test  and  Field  Maturation 

Field  maturation  is  essential  to  achieve  inherent  diagnos¬ 
tics  potential.  When  a  system  is  first  fielded,  it  is  common 
to  find  that  not  all  the  hardware  and  software  provisions  of 
the  diagnostics  have  been  fully  implemented.  In  addition,  the 
operational  use  patterns  and  the  environment  produce  new  failure 
modes  and  diagnostics  indications.  These  new  indications, 
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which  the  BIT  may  not  deal  with  properly,  are  resolved  by  the 
judgment  of  operators  and  maintainers  (who  may  not  have  been 
trained  to  deal  with  them)  with  the  aid  of  technical  data 
(which  may  not  have  been  developed  to  address  them).  A  struc¬ 
tured  diagnostics  maturation  effort  is  the  only  way  most  experts 
see  to  bring  the  diagnostic  capability  to  its  full  potential. 

The  APG-65  and  APG-66  programs  are  excellent  examples  of  effec¬ 
tive  BIT  maturation.  Figure  15  indicates  the  rate  of  diagnos¬ 
tics  growth  of  the  APG-66  radar  during  the  FSD/product ion 
phases.  The  key  features  of  these  programs  should  be  used  in 
structuring  future  maturation  efforts  for  complex  equipment. 


%  ST/ BIT 
Complete 


FIGURE  15.  APG-66  Radar  ST/BIT  Growth 
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The  key  features  of  such  a  diagnostics  maturation  program 
are  as  follows: 

Planning:  The  program  office,  prime  and  subcontractors,  the 

user,  the  operational  test  agencies,  must  accept  the  fact  that 
complex  systems  require  diagnostics  maturation  in  the  operational 
environment.  This  recognition  must  be  coupled  with  commitment, 
funding  and  a  management  plan  to  pursue  diagnostics  maturation 
until  a  mature  capability  is  clearly  demonstrated  in  the  intended 
operational  environment.  The  schedule  for  this  effort  is  dependent 
on  many  factors.  The  APG-66  took  20  months  to  grow  to  full  test 
implementation  and  was  followed  by  another  24  months  of  maturation. 
The  APG-65  schedule  spans  34  months. 

Data  Collection:  A  special  diagnostics  data  collection  and 

analysis  system  is  required  to  capture  information  on  failure 

occurrences  and  causes  in  enough  detail  to  provide  a  credible 

data  base  for  developing  and  implementing  engineering  solutions. 

The  Navy  3M  and  the  Air  Force  MDCSs  are  not  sufficient  but  can  be 

useful  for  this  purpose.  The  inputs  needed  include  the  specific 

/ 

failure  indications  and  circumstances  logged  by  the  aircrew/operator 
a  BIT  debriefing  for  all  missions,  the  specific  indications  of 
detection  and  methods  of  isolation,  a  specific  serial  number 
track  of  LRUs/WRAs  through  the  ultimate  repair  action  and  subsequent 
performance  after  repair,  as  well  as  the  elapsed  time  for  each 
element  of  the  maintenance  event  (set-up,  troubleshoot,  repair, 
verification,  teardown).  Analysis  can  then  focus  on  the  causes 
of  alarms,  CNDs,  BCSs,  no  defects  and  lengthy  repair  times. 

Recorders :  If  the  system  does  not  have  built-in  capability 

to  capture  the  detailed  environmental  condition  information  at 
the  time  of  failure  indication,  additional  sensors  and  recorders 
should  be  installed  on  the  system  during  the  maturation  phase  to 
provide  this  information. 
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Engineering  Manpower;  Knowledgeable  design  engineering  person¬ 
nel  are  essential  at  operational  locations  to  observe  and  analyze 
the  performance  of  diagnostics  capability  so  that  problems  can  be 
recognized  quickly  in  context  with  the  operational  environmental 
variab les . 

Maintenance  Manpower:  A  team  of  operational  maintenance 
personnel  (user,  supporter,  tester)  must  be  available  and  moti¬ 
vated  to  mature  the  diagnostics  and  to  institutionalize  preferred 
troubleshooting  procedures  and  maintenance  policy. 

Operational  Support:  The  operational  entity  which  is  employ¬ 
ing  the  new  system  must  support  diagnostics  maturation.  When  a 
system  fails  or  exhibits  a  diagnostic  problem  the  emphasis  should 
be  on  understanding  the  cause  of  the  problem  rather  than  hurrying 
the  system  back  into  commission.  Operational  units  designated 
for  this  activity  must  have  sufficient  assets  assigned  to  meet 
both  operational  and  maturation  requirements. 

Data  Base  Resources:  Computer  resources  (time,  access,  pro¬ 
gramming)  must  be  available  at  the  operational  location  to  main¬ 
tain  a  diagnostics  maturation  data  base  and  to  support  timely 
analysis. 

Software  Discipline:  Since  many  BIT  anomalies  are  corrected 
by  software  changes,  a  vigorous  software  data  collection  and 
tracking  effort  is  required  to  update  and  control  software 
configuration.  Though  this  activity  is  normally  conducted  at  the 
contractor's  facility,  it  must  interface  closely  with  the  field 
maturation  effort.  The  APG-66  went  through  at  least  eight  block- 
configuration  changes  and  one  major  ECP  during  maturation.  The 
magnitude  of  a  typical  software  olock-change  is  exhibited  by 
Fig.  16  for  the  F-16. 
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BIP 

System  Performance 
Patches 

ST/BIT 

Patches 

Block  4A 

157  Patches 

42  Patches 

Block  4B 

31  Patches 

9  Patches 

Block  4C 

25  Patches 

7  Patches 

Block  4D 

17  Patches 

9  Patches 

Block  4E 

85  Patches 

34  Patches 

Block  4F 

16  Patches 

10  Patches 

Block  4G 

0  Patches 

3  Patches 

Block  4H 

32  Patches 

6  Patches 

Block  41 

35  Patches 

11  Patches 

Block  4J 

7  Patches 

3  Patches 

Subtotal  405  Subtotal  134 

Total  539 


FIGURE  16.  FSD  Software  Modifications  and  Block  Update  Summary 
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Contractor  Support  Base;  Contractor  resources  (time,  engineer 
ing  manpower,  and  system  integration  laboratory/software  support 
facilities)  are  required  to  support  the  field  engineering  activity. 
These  resources  are  necessary  for  verifying  field  anomalies  as 
well  as  formulating,  testing  and  implementing  diagnostic  corrective 
actions.  Production  testing  activities  at  the  factory  may  present 
other  opportunities  for  observing  additional  failure  modes  and 
diagnostics  indications.  These  opportunities  for  diagnostics 
improvements  should  be  used  to  supplement  the  field  maturation 
activity. 


Contractor  Use  of  Diagnostics:  The  contractor  must  be  re¬ 
quired  to  use  the  BIT,  diagnostics  procedures,  and  technical  data 
being  developed  for  Service  use  whenever  he  is  performing  mainten¬ 
ance;  for  example,  during  early  flight  testing  and  interim  contrac¬ 
tor  support.  These  events  present  early  opportunities  for 
maturation,  even  before  operational  testing  starts. 


c.  An  Approach  to  Planning  Future  Avionic  Diagnostics:  In 
addition  to  the  above  activities.  Appendix  E  includes  an  illustra¬ 
tion  of  how  a  system  might  be  structured  to  achieve  significant 
diagnostic  improvement.  This  approach  is  oriented  specifically 
toward  avionics,  but  the  thought  process  should  be  useful  for 
other  applications  as  well.  Regardless  of  the  system  type,  diag¬ 
nostics  capability  must  be  considered  as  a  fundamental  concern 
in  the  conceptual  phase  of  system  architecture  development. 

In  the  world  of  avionics  diagnostics,  bold  steps  are  neces¬ 
sary  to  improve  performance  in  the  field  radically  and  reduce 
substantially  the  cost  of  maintenance.  Support ability  improve¬ 
ments,  particularly  the  contributions  of  avionics  diagnostics, 
require  new  approaches  to  solve  the  problems  faced  by  the  Ser¬ 
vices  in  the  field  today.  Technology  improvements  appear  to 
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offer  the  opportunity  to  make  strides  toward  such  improvement 
Advanced  architectures  provide  the  means  to  achieve  improved 
support  abili  ty . 


d.  Action :  An  agency  should  be  designated  to  be  respon¬ 
sible  for  developing  a  structured  process  for  carrying  diagnos¬ 
tics  through  from  stating  requirements  to  design,  development, 
test,  and  maturation.  The  most  natural  vehicle  for  this  would 
be  a  diagnostic  standardization  program  similar  to  those  started 
for  reliability  in  the  last  10-15  year  time  period  at  both  DoD 
and  NASA.  In  addition,  there  is  need  to  establish  an  R&D  program 
to  develop  the  technologies  required  to  solve  current  problems 
(false  alarm  and  unnecessary  removals). 

B.  INFORMATION 

1.  Conclusion 


Accurate  and  detailed  engineering-quality  information  on 
system  and  component  failures  must  be  provided  for  identifying 
and  solving  equipment  problems  and  focusing  technology  efforts. 

2.  Observations 


A  thorough  understanding  of  how  each  military  Service  operates 
and  a  comprehensive  investigation  and  analysis  of  the  sources  of 
data  are  necessary  if  one  is  to  avoid  misinterpreting  reported 
data.  Wide  variations  in  results,  using  data  obtained  from 
the  same  data  base  and  reported  in  numerous  studies  and  briefings, 
have  been  observed  in  the  course  of  this  study.  Examples  are 
provided  in  Appendix  G.  Based  on  analysis  of  these  cases,  the 
following  findings  were  made. 
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2.  Findings : 

The  planned  operational  concepts,  as  reflected  in  the 
Service  Year-2000  studies,  and  the  complexity  of  current  and 
planned  weapon  systems,  make  it  essential  that  data  systems  be 
capable  of  supporting  units  in  combat  as  well  as  peacetime, 
and  be  user  oriented  (e.g..  Automated  Data  Entry)  for  accuracy 
and  speed.  The  Army  SDC  system  provides  a  reasonable  solution 
for  management  uses.  The  Navy  (Air)  has  an  excellent,  very 
flexible  query  system.  The  Air  Force  AMS  for  C-5As  is  a  superb 
system  but  currently  only  for  C-5As.  The  Navy  data  system  for 
submarines  is  very  comprehens ive  and  has  paid  for  itself  many 
times  over;  for  example,  providing  engineering  quality  data  to 
support  decisions  for  the  extension  of  overhaul  periods.  Addi¬ 
tionally,  all  of  the  Services  have  good  data  systems  which  provide 
excellent  data  during  the  acquisition  process.  In  many  cases,  how¬ 
ever,  the  systems  are  not  imposed  on  the  contractor  nor  continued 
after  the  system  is  fielded. 

The  current  institutionalized  data  systems  are  archaic. 

These  systems  are  useful  only  to  track  trends  and  only  then  when 
(a)  no  significant  data  systems  changes  have  occurred  within  the 
trending  period  and  (b)  no  significant  changes  in  operational 
scenarios/mission  profiles  have  occurred  in  the  trending  period. 
Data  system  usage  in  studies  like  this  one  are  fraught  with 
problems  because  of  the  desire  for  quantitative  backup  for  proposed 
changes/concepts,  hence  the  requirement  to  use  the  data  in  ways 
that  exceed  the  systems  capabilities  which  leads  to  judgment  based 
on  incremental  differences  as  well  as  other  doubtful  uses. 

The  current  military  data  systems  do  not  provide  useful  data 
that  can  be  used  to  characterize  the  R&M  performance  of  a  given 
technology.  For  example,  if  one  were  to  try  to  evaluate  the 
impact  of  changes  from  currently  fielded  solid-state  equipments 
to  VHSIC  implementations,  it  soon  becomes  rapidly  apparent  that 
the  data  base  does  not  permit  one  to  obtain  RfcM  data  at  the 
device  level,  or  by  device  type. 
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Contractors  and  customers  alike  need  engineering  quality 
data  in  order  to  be  able  to  correctly  identify  a  problem  and 
evaluate  candidate  solutions  from  both  a  cost  and  performance 
viewpoint.  Additionally,  it  is  desirable  to  be  able  to  monitor 
performance  of  an  item  by  serial  number  identity,  so  that  the 
effectiveness  of  changes  under  the  configuration  control  system 
can  be  evaluated.  Under  the  current  maintenance  data  systems 
this  is  difficult,  since  the  data  systems  were  not  designed  to 
provide  such  information. 

The  lack  of  good,  easily  accessible  data  to  support  the  re¬ 
quirements  process  is  a  key  reason  for  the  shortfalls  in  the 
requirements  process  and  can  lead  to  unbounded  optimism  and 
very  little  useful  input  to  the  design  process.  Requirements 
documents  for  Service  data  systems  do  not  include  requirements 
for  the  kinds  of  data  needed  to  support  the  acquisition  (especially 


the  R&M  requirements)  process.  In  most  cases,  the  same  kinds  of 
data  elements  are  required  for  yearly  resource  requirements  calcu¬ 
lations,  e.g.,  replenishment  spares,  modification  requirements, 
manpower,  support  equipment,  etc.  The  case  studies  stressed  the 
need  for  reasonable  yet  challenging  R&M  requirements.  In  order 
to  accomplish  this  task,  it  is  necessary  to  know  the  R&M  perform¬ 
ance  of  similar  systems  in  the  field,  differences  in  the  intended 
operational  environment  (mission  profiles)  versus  the  comparable 
systems,  the  potential  impact  of  new  or  different  technology, 
and  how  to  translate  field  R&M  parameters  into  useful  inputs 
for  the  design  process. 

Service  data  systems  have  been  used  as  aids  for  this  process. 
The  F-16  and  F/A-18  programs  both  used  comparability  analysis  to 
estimate  requirements  based  on  similar  systems.  One  problem  with 
this  procedure  relates  to  the  previous  discussion.  The  compara¬ 
bility  analysis,  by  its  very  nature,  usually  presumes  the  status 
quo  for  any  significant  problems  incumbent  in  the  baseline  system, 
e.g.,  masking  real  reliability  and  maintainability  values  due  to 
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inadequacy  in  the  support  system  (training,  support  equipment, 
diagnostics,  etc.).  There  are  notable  examples  where  attempts  to 
unmask  these  problems  have  occurred,  e.g.,  estimated  changes  to  F-16 
radar  removal  rates  if  diagnostics-related  false  removals  could  be 
reduced  to  two  percent.  Detailed  analysis  of  this  type  is  not  easy 
due  to  data  inaccuracy  and  data  availability.  The  data  systems 
also  fall  short  when  it  comes  to  assessing  the  impact  of  technology. 
They  do  not  possess  the  accuracy  and  detail  necessary  to  support 
such  analysis. 

Very  detailed  and  often  meticulous  data  collection  systems, 
far  beyond  the  capabilities  of  standard  data  systems,  are  required 
in  order  to  provide  the  detailed  data  necessary  to  find  and  fix 
diagnostics  problems.  Current  field  data  systems  do  not  provide 
meaningful  data  that  can  be  used  to  assess  the  performance  of 
diagnostics  systems  (BIT,  FIT,  ST).  Again,  as  in  the  case  of 
hardware  problem  reporting,  the  existing  data  systems  at  best 
give  top-level  indicators,  but  no  detailed  information  that  can 
be  used  to  assess  the  diagnostics  performance — let  alone  any  clues 
as  to  the  real  problem  or  root  cause  of  failure — the  information 
needed  by  engineering.  In  most  cases  on-board  recording  or 
monitoring  of  fault  detection  parameters  is  necessary  in  order 
to  find,  fix  and  verify  problems  such  as  intermittent  failure 
indications. 

There  have  been  a  number  of  independent  short-term  efforts 
to  assess  diagnostics  capabilities  on  the  F-15,  F-16  and  F/A-18 
FOT&E/MOT&E  programs,  but  these  have  relied  on  special  data 
collection  systems,  each  tailored  to  the  specific  needs  of  the 
program.  The  F/A-18  program  includes  a  special  (two-year)  BIT 
maturation  program  during  which  special  data  collection  require¬ 
ments  will  be  implemented. 

Detailed,  accurate  data  during  the  early  operational  phase 
of  a  weapon  system  program  is  critical  to  improving  reliability, 
maintainabili ty  and  readiness.  The  importance  of  R&M  up-front 
in  the  design  process  cannot  be  overemphasized.  The  basic  attention 
to,  and  stressing  of,  R&M  principles  allows  the  system  to  start 
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from  a  reasonable  posture.  The  front-end  work  does  not,  however, 
negate  the  need  for  a  planned  growth  maturation  phase;  it  lays 
the  necessary  foundation  for  growth  and  maturation.  It  is 
unreasonable  to  expect  that  all  field  problems  are  or  can  be 
found  and  fixed  prior  to  operational  use  of  the  system.  Diag¬ 
nostics,  again,  is  the  toughest  area  to  mature  due  to  the  inter¬ 
actions  with  all  aspects  of  the  support  structure.  Reliability 
is  but  one  driver  of  diagnostics.  Manpower,  human  factors,  training, 
technical  data,  and  support  equipment  all  interact  with  diagnostics 
to  increase  the  magnitude  of  the  find  and  fix  process.  Unfortunately 
many  of  these  elements  are  not  available  for  a  total  system 
assessment  until  the  early  fielding  phase.  Again,  the  limitations 
of  normal  service  or  contractor  data  systems  make  it  necessary 
to  add  special  adjuncts  to  the  data  systems. 

The  criticality  of  detailed  accurate  data  in  the  early 
operational  phase  as  well  as  the  more  broadly  defined  development 
phase  cannot  be  overemphasized.  In  order  to  provide  the  types 
of  data  necessary  to  identify  and  fix  early  integration  problems, 
operational  reliability  failure  modes,  fault  detection  anomalies, 
fault  isolation  problems,  support  equipment  interface  problems, 
need  for  additional  technical  orders  and  troubleshooting  aids, 
need  for  additional  training,  etc.,  adjuncts  to  Service  data 
systems  such  as  those  listed  in  Fig.  17  may  be  required. 

In  all  cases  the  good  data  systems,  discussed  further  in 
Appendix  G,  have  one  set  of  common  attributes — they  are  designed 
and  implemented  so  that  the  data  are  rapidly  available  and  very 
useful  to  the  person  who  records/reports  the  data.  Given  the 
off-the-shelf  capabilities  available  today  in  information  and 
communications  technology  and  the  extremely  high  payoff  in  readi¬ 
ness  improvements,  weapon  system  quality,  productivity,  O&S  cost 
reductions,  etc.,  demonstrated  at  Dover  AFB,  in  the  Navy  submarine 
program,  and  by  the  Army  SDC  system,  it  is  imperative  that  the 
highest  of  priorities  be  placed  on  developing  and  fielding  state- 
of-the-art  Service  logistics  data  systems  as  soon  as  practical. 

In  Summary,  contractors  and  customers  require  engineering- 
quality  data  in  order  to  be  able  to  correctly  identify  a  problem 
822/1-37 


IV-37 


‘\\  7-  .  VvVVn  •  •  /  .  I  V 


V> 


rwyjrr'wy's  ws  «.>  y  ^  \n  ■.*  '.^  '•*  t  '.■v^ 


ADJUNCTS  TO  SERVICE 
DATA  SYSTEM  CAPABILITY 

Serial  number  tracking 


Tracking  capability  for 
parametric  BIT  detection 
data 


Provisions  for  incorpora¬ 
tion/use  of  time-of-fail- 
ure/failure  indication 


Provisions  for  tracking 
software  related  R&M 
problems 

Special  assessment  team 
including  program  office , 
test  or  user  and  contractor 
design/engineering  support 
personnel 


USE/FUNCTION 

•  Tracking  of  equipment  confi dura¬ 
tion 

•  Failure  analysis 

•  Modification  tracking 

•  Identification  of  bad  actor 
equipment /components 

•  Locate  false  alarm  problems 

•  Identify  intermittent  failure 
problems 

•  Provide  correlation  information 
for  fault  detection  and  fault 
isolation 

•  Aid  in  tracking  of  vertical 
testability,  i.e.,  system  vs. 
intermediate  and/or  depot  level 
test  equipment 

•  Correlation  of  failures  or 
indication  to  other  factors,  e.g., 
temperature,  altitude,  vibration, 
turn-on,  etc. 

•  Provide  for  sorting  software 
related  problems  from  other 
types  of  R&M  problems 

•  User-contractor-SPO  interface 
and  contact  with  system  design 
personnel  to  ensure  the  correct 
problems  are  identified, 
analyzed  and  fixed. 


FIGURE  17.  Adjuncts  to  Service  Data  Systems 
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and  evaluate  candidate  solutions  from  both  cost  and  performance 
viewpoints.  Additionally,  it  is  desirable  to  be  able  to  monitor 
performance  of  an  item  by  serial-number  identity,  so  that  the 
effectiveness  of  changes  under  the  configuration  control  system 
can  be  evaluated.  Under  the  current  maintenance  data  systems, 
this  is  not  possible  since  the  data  system  was  not  designed  to 
provide  such  information. 

Detailed,  accurate  data  collected  during  the  early  opera¬ 
tional  phase  of  a  weapon  system  program  is  critical  to  improv¬ 
ing  reliability,  maintainability,  and  readiness.  It  is  unrea¬ 
sonable  to  expect  that  all  field  problems  are,  or  can  be,  found 
and  fixed  prior  to  operational  use  of  the  system.  Diagnostics, 
again,  is  the  toughest  area  to  mature  due  to  the  interactions 
with  all  aspects  of  the  support  structure.  Reliability  is  but 
one  driver  of  diagnostics.  Manpower,  human  factors,  training, 
technical  data,  and  support  equipment  all  interact  with  diagnos¬ 
tics  to  increase  the  magnitude  of  the  find-and-fix  process. 
Unfortunately,  the  impacts  of  many  of  these  elements  cannot  be 
assessed  until  the  early  fielding  phase. 

3.  Action 

Develop  and  implement  policy  to  ensure  that  funding  and 
procedures  are  in  place  to  get  engineering  quality  data  to 
support  the  planned  maturation  on  specific  programs  which 
include  R&M  growth. 

C.  EDUCATION 

1.  Conclusion 

Actions  to  enhance  and  expand  R&M  knowledge  and  experience 
of  DoD  and  industry  engineers  and  managers  must  be  taken  to 
achieve  long-term  improvement  for  the  full-range  of  weapon  system 
acquisitions . 
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2 .  Observations 


The  cases  studied  showed  that  contractors  do  respond  to 
perceived  DoD  priorities.  One  factor  which  contributed  to 
contractor  perception  of  the  importance  that  DoD  placed  on  R&M 
was  the  capability  and  knowledge  of  personnel  the  contractors 
interfaced  with  on  R&M-related  items.  R&M  have  not  always  been 
given  proper  emphasis  by  engineers,  support  personnel,  and 
managers.  Managers  and  engineers  must  understand  what  the 
different  elements  of  an  R&M  program  are,  how  they  are  inter¬ 
related,  and  what  they  contribute  to  R&M  success.  Additionally, 
DoD  engineers  and  management  would  benefit  from  having  access 
to  highly  qualified,  experienced  personnel  who  could  assist 
them  at  critical  times  during  program  development. 

3.  Findings 

There  are  several  facets  to  the  solution  of  this  problem. 

One  of  these  is  R&M  training.  R&M  training  is  provided  currently 
by  a  number  of  separate  DoD  Service  Schools,  contractors,  and 
educational  institutions,  but  it  is  fragmented  and  limited  in 
scope.  There  is  a  real  need  to  upgrade  R&M  training  throughout 
DoD  and  industry. 

The  need  for  improved  training  methods  results  both  from 
the  fact  that  there  is  little  formal  academic  means  to  obtain 
the  basics  of  reliability  and  maintainability  skills  and  that 
many,  if  not  most,  of  the  current  DoD  education  programs  do  not 
provide  adequate  coverage  of  the  basics  necessary  for  a  success¬ 
ful  R&M  program.  Within  the  case  studies,  there  was  an  apparent 
correlation  between  the  assignment  of  experienced  personnel  and 
the  relative  success  of  the  particular  program.  How  this  ex¬ 
perience  was  gained  was  not  examined. 
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Another  facet  in  the  solution  to  this  problem  is  the  estab¬ 
lishment  of  a  method  of  providing  highly  qualified,  experienced 
"consultants"  to  the  engineers  and  managers  at  critical  times  in 
the  development  cycle.  Various  attempts  have  been  tried  by  the 
Services,  frcm  the  review  method  such  as  the  Navy  Pre-Production 
Reliability  Design  Review  ( PRDR)  and  Air  Force  Independent  R&M 
Reviews  to  more  formal  assistance  teams.  When  experience  is 
limited,  efforts  must  be  made  to  share  it  among  programs. 

One  conclusion  is  that  the  content  and  results  of  the  case 
studies  could  be  used  to  materially  enrich  existing  educational 
programs  within  the  DoD.  This  would  not  only  improve  the  skills 
of  the  R&M  and  engineering  practitioners,  but  also  form  a  basis 
for  educating  program  managers  and  acquisition  management  execu¬ 
tives  in  the  ramifications  and  implications  of  the  various 
alternative  structures  for  a  successful  R&M  program. 

a.  Improving  the  R&M  Capability  in  DoD.  The  current  DoD 
work  force  has  not  received  sufficient  training  or  support  in  R&M 
Under  current  circumstances  it  does  not  appear  likely  that  this 
condition  will  change  in  the  future.  In  fact,  the  problem  is 
likely  to  become  more  serious  as  technology  becomes  more  advanced 

Fundamental  needs  for  improving  the  DoD  R&M  work  force  com¬ 
petence  and  performance  are  divided  into  four  categories — devel¬ 
opment  of  DoD  work  force  capability,  development  of  an  in-house 
advice  and  assistance  capability,  improvement  of  contractor  re¬ 
lations,  and  interface  with  the  academic  community. 

(1).  DoD  Work  Force  Capability.  Development  needs  for 
the  DoD  work  force  capability  for  R&M  fall  into 
three  general  categories.  These  categories  are 
training  of  engineers  and  scientists,  training  for 
personnel  in  ancillary  functional  areas,  and  mana¬ 
gerial  training. 
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There  is  a  critical  need  to  upgrade  the  compe¬ 
tence  of  the  DOD  military  and  civilian  work  force 
in  the  design  aspects  of  R&M.  Existing  training 
programs  emphasize  programmatic  or  mathematical 
aspects  of  R&M  and  have  too  little  coverage  of 
those  design  engineering  techniques  that  have  an 
impact  on  the  actual  R&M  of  the  equipment.  Existing 
quantitative  training  is  patterned  largely  after 
the  standard  quantitative  textbooks  that  tend  to 
be  theoretical  rather  than  practical.  The  DoD  R&M 
work  force  should  be  provided  quantitative  skills 
which  are  relevant  to  the  R&M  functions  performed. 

R&M  training  should  incorporate  appropriate  engi¬ 
neering  methods  and  techniques.  The  engineering 
world  is  one  of  constant  change.  Therefore,  the 
DoD  R&M  training  program  must  be  dynamic  and  include 
courses  in  development  of  system  R&M  requirements;  man¬ 
agement  of  the  acquisition  process  to  optimize  system 
effectiveness  and  readiness;  R&M  program  management; 

R&M  engineering  management;  design  for  R&M;  production 
R&M  assurance;  software  impact  on  R&M;  and  R&M  testing 
and  evaluation. 

There  is  also  a  critical  need  for  managerial 
training  in  such  areas  as  reliability,  maintain¬ 
ability  and  readiness.  R&M  training  for  managers 
can  enhance  the  managers'  decisionmaking  capability. 

In  addition,  there  would  be  a  better  understanding 
between  design  engineering  and  other  management 
decisionmaking  groups. 

(2).  In-House  R&M  Advice  and  Assistance  Capability. 

Within  the  DoD  there  are  broad  ranges  of  experi¬ 
ences  from  the  "school  of  hard  knocks"  to  successes. 
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Attempts  to  capture  this  experience  have  at  best 
been  superficial.  The  fact  that  attempts  have  been 
made  demonstrates  recognition  of  the  need  for 
consolidated  in-house  consultations  and  assistance 
capability. 

Operational  reliability  and  maintainability 
problems  that  are  not  solved  at  the  source  surface 
occasionally  throughout  the  life-cycle.  An  in- 
house  cadre  of  R&M  engineering  specialists  (a  quick- 
reaction  team),  that  can  be  tasked  on  short  notice 
to  provide  advice  and  assistance  or  independent 
opinions  on  day-to-day  operational  problems,  is  a 
possible  solution  worth  exploring. 

(3).  Government/Contractor  Relations.  Case  study  reviews 
have  shown  that  best  results  were  obtained  through 
the  mutual  respect  of  government/contractor  personnel 
working  in  a  cooperative,  non-adversarial  manner. 
Likewise,  within  the  government  environment,  similar 
cooperation  and  the  ability  to  assess  R&M  activity 
are  important  contributors  to  improved  R&M. 

Most  of  the  actual  R&M  engineering  work 
for  DoD  is  performed  by  contractor  personnel. 
Government  engineering  personnel  most  frequently 
concentrate  on:  (1)  developing,  reviewing  and 
assessing  documents  that  prescribe  R&M  requirements 
to  contractor  engineers;  and  (2)  reviewing  the 
results  of  contractor  R&M  engineering  activity. 

It  is  imperative  that  Government  engineering  and 
technical  personnel  be  thoroughly  versed  in  R&M 
engineering  needs  and  related  technology.  Achieve¬ 
ment  of  technical  competence  and  respect  is  essen¬ 
tial  to  fostering  cooperation  between  Government 
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and  contractor  personnel  and  in  evaluating  the 
contractor's  R&M  plans. 


(4).  Academic  Relations.  Most  college  and  university 
engineering  curricula  concentrate  on  basic  skills 
and  disciplined  knowledge.  Only  a  few  courses 
address  R&M  technology.  There  is  a  need  for  in¬ 
creased  communication  between  the  academic  community 
and  the  R&M  engineers  and  technicians.  It  is 
unlikely  in  the  foreseeable  future  that  fully 
adequate  training  and  education  in  R&M  technology 
will  be  widely  available  from  the  academic 
community. 

b.  Current  Status  of  R&M  Training  and  Education  in  DoD. 

There  are  several  education  and  training  sources  from  which  DoD 
R&M  engineers  may  obtain  or  may  have  been  obtaining  job-enhancing 
technical  and  management  skills,  knowledge,  and  abilities. 

(1).  Academia.  Formal  college  and  university  training 
specifically  applicable  to  the  needs  of  the  DoD  R&M  work  force 
is  quite  limited.  However,  some  educational  opportunities  are 
available  to  the  R&M  work  force.  Some  DoD  installations  have 
access  to  local  university /col lege  courses  which  will  help  to 
develop  R&M-related  skills.  Though  specific  offerings  vary 
between  installations,  individual  courses  and/or  educational 
programs  provide  education  in  R&M  subjects,  such  as  reliability, 
system  management,  contracts  and  procurement.  The  education 
available  is  generally  introductory  in  scope  and  does  not  provide 
the  opportunity  for  in-depth  education  in  R&M  engineering 
techn iques. 
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(2).  DoD  Technical  and  Professional  Training,  Within  the 
Department  of  Defense ,  there  are  a  number  of  schools  that  focus 
on  the  technical  problem  of  acquiring  new  weapon  systems. 


•  Defense  Systems  Management  College  (DSMC).  The 
current  curriculum  of  DSMC  includes  the  20-week 
Program  Manager's  Course,  a  3-week  Executive 
Refresher  Course,  and  a  1-week  Flag-Rank 
Refresher.  These  courses  include  R&M  and 
readiness  issues  in  the  general  course  of 
instruction  as  elements  of  the  major  case 
studies  and  class  exercises.  Guest  lectures 

by  noted  authorities  in  the  field  are  also 
utilized.  This  approach  places  the  subjects 
into  context  and  allows  the  students  to  grasp 
the  complexities  and  interrelationships  of 
of  the  various  issues  instead  of  viewing  the 
subjects  in  isolation.  The  current  approach 
may  benefit  from  examining  the  material  that 
has  been  gathered  by  the  case  studies  and 
using  this  material  to  update  and  expand  the 
content  of  the  course  case  studies  and  exercises 
used  at  DSMC. 

•  U.S.  Army  Management  Engineering  Training  Activit 
( US AM ETA) .  USAMETA  is  a  management  training, 
research,  and  consulting  organization  within  the 
U.S.  Army  Materiel  Development  and  Readiness 
Command  (DARCOM)  and  trains  13,000  DoD  students 
per  year  in  short,  concentrated  management  and 
engineering  courses.  The  current  curriculum 

of  100  courses  includes  eight  which  are  designed 
to  satisfy  expressed  training  needs  for  the 


R&N  engineering  community  within  DoD.  Several 
other  courses  contain  R&M  subject  material. 
These  courses  are  designed  for  R&M  support 
personnel  and  managers  from  other  functional 
areas. 


Army  Logistics  Management  Center  ( ALMC) .  ALMC 
is  a  DARCOM  school  whose  mission  is  to  conduct 
training,  perform  research,  formulate  doctrine, 
and  provide  information  and  consulting  programs 
for  logistics  management  matters.  The  ALMC 
curriculum  consists  of  71  courses.  These 
courses  are  designed  for  journeymen  and  man¬ 
agers.  With  the  exception  of  a  specialized 
Army-peculiar  reliability-centered  maintenance 
course,  the  ALMC  curriculum  includes  no  R&M 
courses.  However,  many  courses  include  blocks 
of  training  on  R&M.  The  DARCOM  Intern  Training 
Center  (ITC)  located  at  Texarkana,  Texas,  is 
a  component  of  ALMC.  ITC  provides  R&M  training 
for  two  one-year  DARCOM  Intern  Training  Pro¬ 
grams — the  Maintainability  Engineering  Intern 
Program  and  the  Quality  and  Reliability  Engi¬ 
neering  Intern  Program.  These  programs  are 
designed  to  assist  graduate  engineers  in 
making  the  transition  into  the  Army  R&M 
community. 

Navy  Acquisition  Logistics  Management  School. 
The  school  educates  Navy  military  and  civilian 
personnel  on  the  current  acquisition  and  logis¬ 
tics  policies  and  procedures.  Each  one-week 
class  consists  of  15-40  program  management, 
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fiscal  and  support  personnel.  One  session  is 
devoted  to  R&M  and  includes  an  overview  of  this 
R&M  study. 
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Air  Force  Institute  of  Technology  (AFIT),  School 


of  Systems  and  Logistics.  AFIT  conducts  courses 


that  are  designed  to  provide  instruction  in 
systems  logistics  and  management  areas  related 
to  military  and  civilian  duty  assignments. 

The  Professional  Continuing  Education  ( PCE ) 
Program  consists  of  approximately  58  courses 
of  relatively  short  duration — 1  to  7  weeks. 
Course  content  generally  emphasizes  the 
operational  areas  of  systems  acquisition  and 
logistics  management.  As  a  part  of  the  above 
curriculum,  AFIT  offers  four  R&M  courses  for 
Air  Force  students.  The  courses  address 
reliability,  R&M  research  and  applications, 
reliability  theory,  and  life-cycle  management. 

Other  Short  Courses.  Other  means  also  used 
by  the  military  services  for  R&M  training 
are  contract  training  and  utilization  of  in- 
house  experts.  The  Defense  Logistics  Agency 
( DLA)  and  the  Navy  make  extensive  use  of  this 
approach.  Such  courses  are  usually  intended 
to  be  conducted  for  some  limited  time  period 
and/or  specialized  audiences. 
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c.  Action.  DoD  R&M  training  is  currently  being  provided 
by  a  variety  of  sources,  both  in-house  and  contractor.  However, 
there  is  no  organized  program  to  establish  and  maintain  a  coordi¬ 
nated  R&M  curriculum  for  engineers,  support  personnel,  or  managers 
on  a  DoD-wide  basis.  As  a  result,  there  is  opportunity  for  non¬ 
uniformity  in  training,  overlap  in  course  offerings,  and  omissions 
where  training  may  be  badly  needed.  There  is  a  need  to  integrate 
R&M  training  activities  throughout  DoD. 

Two  basic  alternatives  could  meet  the  criteria  specified 
above : 

•  Establish  a  new  organization  with  a  mission  to  provide 
R&M  training  and  related  advice  and  assistance  to  the 
DoD  community. 

•  Assign  executive-agency  responsibility  to  an  existing 
organization  for  R&M  training.  The  executive  agent 
could  work  closely  with  DoD,  academia,  and  industry  train¬ 
ing  institutions  to  develop  a  comprehensive  R&M  training 
curriculum. 

In  addition,  consideration  should  be  given  to  reestablishing 
Master's  Degree  Programs  in  Reliability  at  schools  like  the 
Army's  Red  River  and  the  Air  Force's  Master's  Degree  School, 
which  were  the  source  of  valuable,  well-trained  people. 
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D.  OTHER  OBSERVATIONS 


1 .  The  Discipline  and  Structure  of  Management 

a.  Observations :  The  techniques  of  establishing  requirements 
and  structuring  them  to  achieve  the  desired  fielded  capability 
are  still  not  well -understood  by  management  even  though  they  may 
be  understood  by  the  R&M  practitioners.  Efforts  reviewed  in  this 
study  often  resulted  in  ill-defined  requirements,  or  requirements 
for  unrealistic  performance  levels. 

The  net  results  have  been  requirements  written  with  good 
intentions  but  lacking  in  their  ability  to  provide  a  realistic 
framework  for  design,  development  and  production.  As  a  result, 
priorities  were  not  stabilized  within  the  large  development 
efforts  necessary  to  produce  systems  with  fielded  performance 
that  met  the  original  expectations.  The  simple  fact  that  require¬ 
ments  are  most  often  written  in  contractual  terms  which  are 
structured  so  that  their  achievement  will  result  in  field  relia¬ 
bility  considerably  lower  than  desired  continues  to  escape  many 
involved  in  acquisition  and  program  management. 

The  need  to  improve  the  nature  of  the  requirements  statement 
and  its  translation  into  contractual  terms  and  thence  into  design 
is  highlighted  by  the  complexity  of  modern  systems  and  the  resul¬ 
tant  drive  toward  more  automation  in  the  diagnostics  area. 

Within  the  built-in-test  discipline,  specified  requirements  of 
98  percent  fault  detection  have  ended  up,  in  some  cases,  in 
addressing  only  one-tenth  of  the  actual  malfunctions  that  the 
system  ultimately  experienced  (see  Fig.  18). 

b.  Findings:  It  is  essential  that  more  attention  be  focused 
on  the  establishment  of  realistic  requirements,  awareness  of  the 
program  implications  to  achieve  these  and  incentives  for  achieve¬ 
ment.  Program  manager  and  chief  engineer  personnel,  as  well  as 
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design,  reliability  and  maintainability  engineers,  must  understand 
the  consequences  of  their  respective  programmatic  actions. 

Within  the  case  study  review  process  two  discrete  high-payoff 
items  were  identified:  establishment  of  requirements  and  the  use 
of  contractor  incentives. 

(1)  Establishment  of  Requirements.  Acquisition  programs 
must  be  structured  early  starting  with  concept  approval  and  leading 
to  a  Full-Scale  Development  contract  containing  realistic  R&M 
requirements  (not  merely  goals),  so  that  R&M  can  compete  effectively 
for  management  support  with  all  other  program  requirements.  When 
R&M  requirements  are  specified  as  goals,  technical  performance 
requirements  will  receive  a  higher  priority  in  the  management 
decision  process.  The  FIREFINDER  case  study  provides  a  good 
example  of  goals  versus  requirements.  The  TPQ-37  Advanced  Devel¬ 
opment  Contract  contained  only  a  250-hour  MTBF  goal  and  it  appears 
that  the  contractor  placed  a  major  emphasis  on  performance  (Ref. 
FIREFINDER  Case  Study,  p.  57/1-23).  When  the  TPQ-37  system  went 
from  Advanced  Development  to  Low-Rate  Initial  Production  (LRIP), 
a  firm  90-hour  MTBF  requirement  was  added  to  the  contract,  along 
with  an  incentivized  reliability  demonstration  test.  The  measured 
MTBF  went  from  33  hours  for  Advanced  Development  configuration 
to  115  hours  for  the  LRIP  radars.  Many  things  contributed  to 
the  reliability  improvement;  however,  it  is  believed  that  the 
management  emphasis  placed  on  meeting  the  reliability  requirements 
during  the  incentivized  demonstration  test  was  a  major  factor. 

The  R&M  requirements  must  have  contract  specified  verification 
to  be  true  requirements.  The  contract  must  specify  the  items  to 
be  measured,  method  of  measurement  and  the  environment  in  which 
the  measurement  will  be  made.  The  planned  R&M  growth  should  be 
a  contractual  requirement  also. 
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Requirements  must  be  specified  within  the  context  of  realistic 
operating  environments.  Establishment  of  a  mission  and  life 
profile  for  the  system  early  in  the  program  is  a  necessary  precursor 
to  meeting  the  desired  levels  for  R&M  in  the  field. 

Mission  profile  development  should  begin  in  the  conceptual 
phase  and  continue  throughout  the  program.  Detailed  design 
analysis  and  decisions  must  be  performed  in  the  context  of  the 
expected  system  environmental  conditions.  If  mission  profile  or 
life  profiles  are  incomplete,  then  critical  design  decisions  may 
not  consider  the  expected  operational  stresses.  Reliability  and 
maintainability  are  direct  functions  of  these  operational  stresses. 

Detailed  design  actions  must  address  the  stresses  that  are 
projected  for  the  total  system  life  profile.  The  non-mission 
portion  of  the  life  profile  of  the  system  will  often  be  a  primary 
design  requirement  as  in  the  case  of  missile  storage  for  extended 
periods  (see  Fig.  19).  The  unique  aspects  of  the  non-mission 
portions  of  the  life  profile  need  to  be  translated  into  design 
requirements.  Profiles  must  address  the  total  usage  including 
dormant  time,  time  waiting  for  the  mission,  and  maintenance 
time . 


(2)  Use  of  Incentives.  Contract  incentives  such  as 


reliability  improvement  warranties  (RIW)  can  increase  contractor 
motivation  to  improve  reliability  and/or  maintainability.  Figure 
20  shows  the  reliability  improvement  for  the  Lightweight  Doppler 
Navigation  System  (LDNS)  observed  in  that  RIW  case  study.  The 
case  studies,  even  though  limited,  indicated  that  use  of  these 
incentives  could  provide  payoff  through  additional  contractor 
actions  for  reliability  growth  and  diagnostic  maturation  programs. 
Other  studies  have  also  confirmed  this  fact.  Figure  21  is  a 
summary  of  data  from  the  Air  Force  Affordable  Acquisition 
Approach  (A^).  The  effect  of  the  period  of  performance  used 
for  assessment  must  be  considered.  Short  duration  measurement 
periods  may  have  only  short-term  effects  and  tend  to  focus  too 
much  attention  on  the  measurement  methods  rather  than  on  the 
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•  Non-Mission  Stresses 

Result  from: 

-  Transportation 

-  Handling 

-  Storage 

-  Test  or  Checkout 


•  Duration  of  Non-Mission  Periods: 


PERCENT  OF 
LIFE  IN 

MISSION  NON-MISSION 

LIFE  TIME  TIME 

Missile  Electronics  10-15  yrs  30  min  99.9% 

Aircraft  Electronics  15  yrs  4,000  hr  96.9% 


FIGURE  19.  Non-Mission  Portions  of  a  System  Life 

Profile  are  Significant  in  Stress  Levels 
and  Duration 
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500  HOUR  GUARANTEED 


Oct  79  Feb  80  Oct  80  Feb  81  Oct  81  Feb  82  Oct  82  Feb  83 

TIME  PERIOD 


FIGURE  20.  Lightweight  Doppler  Navigation  System 
AN/AS N  128  RIW — MTBF  Achieved  (CUM) 


822/1-54 


IV-54 


y, 


•  10  PROGRAMS  WITH  RELIABILITY  INCENTIVES  REVIEWED 

•  TYPES  OF  INCENTIVES 

•  RIW 

•  MTBF  GUARANTEE 

•  AWARD  FEE 

•  RELIABILITY  PERFORMANCE  INCENTIVES 

•  9  OUT  OF  10  FIELDED  SYSTEMS  MET  OR  EXCEEDED 
RELIABILITY  REQUIREMENTS 

•  8  OF  10  SYSTEMS  HAD  INCENTIVES  BASED  ON 
OPERATIONAL  PERFORMANCE 

WHAT  WAS  DONE  DIFFERENTLY 


•  INCREASED  MANAGEMENT  ATTENTION  10  OF  10 

•  RELIABILITY  ACHIEVEMENT  3  OF  10 

PROGRAM  MANAGER  ASSIGNED 

•  INCREASED  DESIGN  ANALYSIS  4  OF  10 

•  ADDITIONAL  TRADE  STUDIES  4  OF  10 

•  MADE  DESIGN  CHANGES  4  OF  10 

•  ADDITIONAL  TESTING  5  OF  10 


FIGURE  21.  R&M  Contract  Incentive 
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actions  required  to  improve  the  long  run  operational  R&M  perfor¬ 
mance  of  the  equipment.  For  example,  an  award  fee  for  passing  a 
reliability  qualification  test  provides  an  incentive  until  the 
test  is  passed,  but  it  will  not  provide  an  incentive  for  the 
subsequent  production  of  reliable  or  maintainable  systems. 

Other  contractual  levers  need  to  be  employed  for  the  production 
phase.  An  RIW  may  be  structured  to  provide  incentives  for  the 
contractor  to  design  and  test,  to  mature  the  operational  system 
(R&M),  and  to  maintain  quality  in  production  because  it  is  also 
in  his  best  interest  to  do  so.  Up-front  design  actions  taken  in 
response  to  an  RIW  will,  however,  be  related  to  the  contractors' 
prediction  of  reliability  risk  and  the  economics  of  making  or 
not  making  the  objective.  An  efficient  RIW  may  be  difficult 
to  negotiate  for  a  sole  source  contract.  RIW's  appear  most 
suitable  for  systems  for  which  depot  repair  is  economical  and 
where  risks  are  bounded. 

It  should  be  noted  that  the  techniques  available  to  provide 
incentives  to  contractors,  e.g.,  awards  fees,  RIWs,  etc.,  will  not 
by  themselves  assure  success.  A  well-thought-out  and  coordinated 
set  of  incentives,  requirements,  program  structure,  test  program, 
and  maturation  are  all  needed.  The  cases  studied  clearly  showed 
that  R&M  incentives,  even  small  ones,  have  impacts  and  that  well- 
structured  R&M  incentives  have  significant  potential. 

Incentive  programs  can  be  structured  to  promote  both  disciplined 
R&M  design  and  development,  to  bridge  the  gap  between  lab  results 
and  field  usage,  and  to  aid  in  the  rapid  correction  of  field 
identified  problems.  Incentives  should  be  tailored  to  individual 
programs,  not  merely  copied  blindly  from  program  to  program. 
Experience  with  incentives  should  be  analyzed  in  context  and 
the  results  of  these  analyses  used  to  educate  appropriate  DoD 
management  to  assure  that  programming  and  budgeting  activities 
include  provisions  for  incentives. 
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2 .  New  Starts  or  Upgrades 

a.  Observations :  If  defense  policies  are  successful  and 
major  hostilities  are  avoided  during  the  next  two  decades,  a  high 
percentage  of  the  Year-2000  inventory  of  equipment  is  already  in 
the  field  today  (see  Fig.  22).  The  DoD  Acquisition  Guidelines 
established  Preplanned  Product  Improvement  (P^I)  as  the  second 
of  31  priorities.  This  in  essence  was  a  proposal  to  keep  the 
fielded  inventory  from  becoming  a  "wasting  asset"  by  structuring 
the  system  design  process  to  accommodate  a  strategy  of  upgrade 
both  to  improve  performance  and  reduce  failures. 


1990 

2000 

AIRLIFT  ONLY 

99% 

83% 

ALL: 

93% 

55% 

BOMBERS 

TANKERS 

STRATEGIC  DEFENSE 

AIRLIFT 

TACTICAL  FIGHTERS 

SOURCE:  AF/XOXF 

FIGURE  22.  Percentage  of  Current  Air  Force  Aircraft 
Expected  to  Still  be  in  Service  in  1990 
and  2000 

b.  Findings ;  One  of  the  new  challenges  proposed  by  a 
disciplined  approach  to  program  structuring  is  recognition  of 
the  fact  that  in  spite  of  the  best  design  and  manufacturing 
efforts  there  still  remain  significant  unknowns  which  can  only 
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be  detected  and  addressed  after  the  system  is  operated  in  the 
field  by  the  actual  user  command.  Recognition  of  this  is  imper¬ 
ative  to  provide  structuring  for  total  growth  and  maturation 
from  early  program  conception  through  the  first  two  or  three 
years  of  fielding  experience.  This  total  growth  concept  must  be 
closely  coupled  with  the  disciplined  approach  discussed  in 
Appendix  B  to  prevent  it  from  degenerating  into  a  practice  of 
to  deferring  today's  problems  until  tomorrow. 


APPENDIX  A 


CASE  STUDIES  ORGANIZATION  AND  PARTICIPANTS 

This  appendix  describes  the  organization  established  to 
treat  the  program  structure  aspects  of  the  study  objective.  In 
addition,  the  case  study  methodology,  selection  process,  and 
program  review  process  are  discussed  and  the  major  participants 
identified . 


A.  OBJECTIVE 


The  overall  study  objective  stated  in  the  Task  Order  is: 

"To  identify  and  provide  support  for  high-payoff 
actions  which  the  DoD  can  take  to  improve  the 
military  system  design,  development  and  support 
process  so  as  to  provide  guantum  improvements  in 
R&N  and  readiness  through  innovative  uses  of 
advancing  technology  and  program  structure." 

From  this  objective,  two  distinct  program  structure-related 
tasks  were  derived: 

(1)  To  assess  the  impact  of  program  structuring 
on  future  DoD  requirements  for  improved  R&M 
and  readiness;  and 

(2)  To  evaluate  the  potential  and  recommend 
strategies  that  might  result  in  quantum 
increases  in  R&M  or  readiness  through 
innovative  use  of  program  structuring. 
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B.  CASE  STUDY  STRUCTURING 


In  structuring  this  portion  of  the  overall  study  activity, 
consideration  was  given  to  the  methodology  to  be  employed,  the 
selection  of  cases  and  the  overall  review  process  to  be  used. 

This  structuring  was  established  by  the  study  executive  council 
in  the  early  phase  of  the  study  and  a  detailed  case  study 
organization  was  instituted,  as  shown  in  Figure  A-l.  Case  study 
results  are  contained  in  IDA  record  documents  (Refs.  A-l  through 
A-6). 

Consideration  was  given  to  breadth  and  depth  of  case  study 
activity.  Goals  were  established  to:  look  across  all  Services 
(Army,  Navy,  and  Air  Force);  encompass  different  sizes  (small  to 
very  large);  look  at  a  range  of  complexities  (relatively  simple 
to  very  complex)  and;  look  at  various  operational  environments 
(truck-mounted,  helicopter,  fighter  aircraft  and  shipboard). 

In  establishing  the  early  approach,  the  executive  council 
decided  that  the  activity  should  be  focused  on  successful  systems 
in  an  attempt  to  glean  the  strengths  of  each  into  an  overall 
composite  structure  which  might  be  implemented  in  the  next  gen¬ 
eration  weapon  system.  By  taking  this  approach  and  establishing 
open  and  candid  exchanges,  it  was  felt  that  people  close  to  the 
program  might  be  more  inclined  tQ  discuss  failures  as  well  as 
successes,  once  they  recognized  that  this  study  was  not  geared 
to  “rock  throwing"  or  "witch  hunting."  This  turned  out  to  be  a 
successful  approach  as  many  avenues  were  opened  on  many  programs. 
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CASE  STUDY  CASE  STUDIES 

DIRECTOR  _______  STEERING 

PAUL  GOREE  — — —  COMMITTEE 
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C.  METHODOLOGY 


In  developing  the  methodology  to  permit  cross-program 
analysis,  data  were  collected  from  the  F-16,  Blackhawk  and  F/A-18 
programs  to  get  an  idea  of  areas  upon  which  to  focus  the  case 
study  activity.  The  top  ten  contributors  to  maintenance  events 
were  as  follows: 


RANKING 

F-16 

ITEM 

TOTAL 

MAINTENANCE  EVENTS 

1 

Engine 

14.9% 

2 

Fire  Control  Radar 

12.5 

3 

Airframe 

12.4 

4 

Landing  Gear 

7.4 

5 

Auxiliary  Power 

6.7 

6 

Fuel  System 

6.2 

7 

Flight  Controls 

5.8 

8 

Crew  Station 

4.1 

9 

Electrical  Power 

3.7 

10 

Lighting 

UH-60A  BLACKHAWK 

3.0 

76.0% 

RANKING 

ITEM 

TOTAL 

MAINTENANCE  EVENTS 

1 

Airframe 

19.5% 

2 

Rotor  Systems 

16.4 

3 

Avionics 

10.9 

4 

Drive  System 

9.2 

5 

Flight  Control  System 

8.2 

6 

Electrical  System 

5.7 

7 

Instrument  System 

5.6 

8 

Landing  Gear  System 

5.1 

9 

Pneudraulics 

4.5 

10 

T700  Engine 

4.2 

89.3% 
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RANKING 


ITEM 


TOTAL  MAINTENANCE  EVENTS 


1 

Weapons  Control 

18.7% 

2 

Landing  Gear 

10.7 

3 

Airframe 

9.0 

4 

Electrical 

5.8 

5 

Fuel  System 

5.8 

6 

Weapons  Delivery 

5.5 

7 

Flight  Controls 

5.1 

8 

FCES 

4.3 

9 

Maintenance  and  BIT  System 

4.2 

10 

Environmental  Control  System 

3.6 

72.7% 


From  the  review  of  the  "Top  10"  lists,  the  focus  centered  on 
radars  and  engines,  with  high  interest  in  fire  control  radars  due 
to  potential  implications  of  the  associated  built-in-test/diagnos¬ 
tics.  In  addition,  it  was  felt  that  some  findings  in  the  radar 
area  could  be  translated  credibly  to  complex  electronics  in 
general. 

The  "Top  10"  lists  pointed  to  another  area  of  interest  as 
well.  Note  that  the  F-16  engine  is  number  one  on  its  list  while 
the  Blackhawk  engine  drops  to  number  ten.  In  an  effort  to  identify 
what  was  done  to  achieve  major  improvements  in  engine  development 
and  to  move  into  the  mechanical  systems  area,  the  T700  engine  was 
added  to  the  list  of  studies. 
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D.  SELECTION  PROCESS 


The  process  of  selecting  systems  for  case  studies  evolved 
over  the  course  of  several  months.  Nominations  for  candidate 
systems  were  accepted  from  both  the  Services  and  industry,  repre 
senting  many  diverse  capabilities.  The  candidates  ranged  from 
the  relatively  simple  AIM-7  and  AIM-9  missiles  to  the  very 
complex  TRIDENT  submarine.  Between  these  extremes  were  engines, 
radars  and  navigation  systems  representing  varying  degrees  of 
complexity. 

After  considerable  discussions  within  the  core  group,  a  de¬ 
cision  was  made  to  do  pilot  case  studies  on  the  airborne  radars 
currently  installed  in  the  F-15,  F-16  and  F/A-18  aircraft.  The 
intentions  were  to  develop  a  case-study  model  using  similar  air¬ 
borne  radar  systems  that  could  then  be  used  to  perform  studies 
on  a  relatively  large  number  of  other  systems.  The  airborne 
radar  systems  seemed  to  be  ideal  choices  for  the  pilot  programs 
because  they  were  high  on  the  list  of  subsystems  that  usually 
cause  major  problems  for  weapon  systems  in  each  Service. 

Initial  efforts  to  develop  case  studies  from  researching 
data  and  literature  sources  yielded  poor  results  at  best  and 
dictated  a  change  in  tactics. 

The  second  effort  to  bring  together  a  useful  case  study 
resulted  in  the  formation  of  working  groups  composed  of  prime 
contractors  and  subcontractors  and  Service  program  office 
representatives.  Bringing  together  the  key  people  associated 
with  the  programs  significantly  improved  efforts  to  capture  what 
was  done  to  make  each  program  successful. 

In  conducting  the  initial  case  studies  it  became  apparent 
that  capturing  data  in  a  format  conducive  to  comparing  programs 
would  be  difficult  even  with  like  systems  because  of  differences 
in  the  way  data  are  preserved  on  various  programs  and  the  manner 
in  which  various  programs  are  managed.  This  fact,  coupled  with 
an  understanding  of  the  amount  of  time  and  effort  required  to 
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do  comprehensive  case  studies,  helped  focus  for  the  remaining 
case-study  activity.  As  a  result,  the  scope  was  narrowed  to 
look  at  radar  systems  of  varying  complexity  (Lightweight  Doppler 
Navigation  System,  Airborne  Fire  Control  Radars,  Weapon  Locating 
Radars,  and  Shipborne  Fire  Control  Radar),  and  one  mechanical 
system,  the  T700  engine.  The  inclusion  of  the  T700  was  important 
for  two  reasons.  First,  it  added  a  mechanical  system  to  the  list 
and  second,  it  provides  a  bridge  to  follow-on  efforts  that  might 
include  additional  mechanical  devices. 

E.  PROGRAM  REVIEW  PROCESS 

R&M  Program  Review  Elements  were  established  early  in  the 
study  program  so  that  case  studies  could  be  conducted  in  a  con¬ 
sistent  and  thorough  manner.  These  elements  provided  a  structure 
for  gathering  information  for  the  case-study  reports.  The  26 
review  elements  were  under  five  major  headings:  contracting, 
management,  design,  manufacturing  and  test  and  evaluation. 

Each  element  was  defined  and  a  list  of  questions  to  be  addressed 
was  prepared. 

Library  research  provided  a  substantial  number  of  reports, 
symposium  papers,  military  standards,  and  other  information 
pertinent  to  the  study  (over  eight-hundred  and  fifty  documents; 

Ref.  IDA  Record  Document  D-18,  "Bibliography"). 

Prime  and  subcontractors  were  visited  and  information  gathered 
through  personal  interviews  with  people  directly  involved  with 
the  program.  In  some  cases,  the  contractor  did  the  bulk  of  the 
case-study  report  work;  in  others,  the  contractor  provided  basic 
inputs  and  the  IDA  R&M  study  group  personnel  assisted  in  the  prep¬ 
aration  of  the  report.  Data  from  the  individual  case  study  reports 
used  in  this  report  were  reviewed  for  accuracy  with  the  contractors 
and  the  military  services.  Contractor  personnel,  involved  in  the 
preparation  of  case  studies,  participated  in  the  IDA  R&M  Study 


Group  Meetings  accompanied  by  representatives  from  the  appropriate 
military  procuring  activity. 

The  case  study  review  approach  was  based  on  building  a  founda¬ 
tion  for  analysis  and  to  analyze  the  front-end  process  of  successful 
programs  for  ways  to  attain  R&M,  mature  it,  and  improve  it.  Concur¬ 
rency  and  resource  implications  were  considered.  Existing  case 
study  reports,  new  case  studies  conducted  specifically  to  document 
quantitative  data  for  cross-program  analysis,  and  documents, 
presentations,  and  other  available  literature  were  used  to  build 
the  analytical  foundation.  In  addition,  focused  studies  for 
specific  technology  implications  were  conducted  by  individual 
technology  working  groups  and  documented  in  their  respective 
reports. 

In  some  areas  where  program  documentation  and  records  did 
not  exist,  the  actual  experience  and  judgment  of  those  involved 
in  the  programs  were  captured  in  the  case  studies.  Likewise,  in 
the  analysis  process,  the  broad  base  of  experience  and  judgment 
of  the  military/industry  executive  council  members  and  other 
participants  was  vital  to  understanding  and  analyzing  areas  where 
specific  detailed  data  were  lacking. 

F.  CASE  STUDY  PARTICIPANTS 

Without  the  detailed  efforts,  energies,  patience  and 
candidness  of  those  intimately  involved  in  the  programs  studied, 
this  effort  would  not  have  been  possible  within  the  time  and 
resources  available,  participants  making  major  contributions  to 
these  case  studies  are  listed  below.  In  addition  to  the  listed 
participants,  candid  comments  and  inputs  were  received  from 
personnel  from  many  other  programs,  from  generals,  admirals,  and 
other  senior  executive  Service  personnel  throughout  the  course  of 
the  study.  Every  effort  has  been  made  to  capture  these  valuable 
lessons  within  the  text  of  this  document. 
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CASE 

Bailey,  E.W. 

Boiles,  J.H. 

Booton,  Bill 
Brooks,  C.R. 

Butler,  Norman 
Cutchis,  P. 
Danielson,  O.F. 
Dobyns,  Dick 
Eikerenkoetter ,  J. 
Fahey,  J.R. 

Farrell,  C.F. 

Fisher,  G. 

Galanti,  Carl 
Gebhardt,  C.C. 
Gibson,  LTC  P.S. 
Goldstein,  S. 

Goree,  Paul  F. 
Griffin,  LTC  Larry  D 
Gunkel,  Dick 
Hatfield,  Phil 
Job,  M.A. 

Johnson,  Marvin 
Kamrass,  Murray 
Kennedy,  P.E. 

Kern,  G.A. 

Koon ,  K . F . 

Kunznick,  G.A. 
Lanctot,  R. 
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Makowsky,  Larry  C. 
McAfee,  Naomi 


STUDY  PARTICIPANTS 

McDonnell  Douglas 
McDonnell  Douglas 
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IDA  Consultant 

IDA  (Case  Study  Director) 
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McIntyre,  Marlene 
Miller,  R.M. 

Mu s son,  Tom 
Nelson,  F.B. 
Osifchin,  Edward 
Pace,  W.E. 

Parham,  David 
Perkins,  C.P. 
Przedpelski,  Z.J. 
Pyle,  Roy 
Quinn,  John  C. 
Rakeman,  J. 

Rogger,  W.R. 
Russell,  Capt.  Bob 
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Selling,  A.L. 
Slinkard,  J. 
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Westinghouse 
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APPENDIX  R 


A  DISCIPLINE  FOR  PLANNING  AND  ANALYZING  PROGRAM  STRUCTURE 

In  the  course  of  conducting  case  studies  for  this  project/  an 
organized  approach  for  analyzing  weapon  system  program  structure 
has  been  developed  and  tested  over  the  duration  of  the  project. 
Though  initially  structured  to  permit  evaluation  and  insight  into 
existing  programs,  perhaps  the  more  important  contribution  could 
come  from  using  this  approach  for  planning  and  analysis  during  the 
formative  stages  of  the  creation  of  weapons  systems  programs, 
as  well  as  a  monitoring  tool  for  management  use  throughout  the 
life  of  the  program. 

The  focus  of  this  process  is  to  understand  reliability,  main¬ 
tainability  and  readiness  implications  of  programmatic  structur¬ 
ing  and  management  decisions.  It  is  based  on  actual  experiences 
which  are  documented  in  detail  in  Appendixes  C  and  D,  the  individual 
case  studies  and  expert  judgment  of  the  many  military  and  industry 
study  participants.  It  encompasses  considerations  for  variations 
in  programs  and  the  acquisition  environment;  structuring  for  R&M 
in  weapon  system  programs;  interrelationships  and  dependencies  of 
program  elements;  and  concurrency  and  scheduling. 

This  disciplined  approach,  if  followed,  could  lead  to  signif¬ 
icant  improvements  in  R&M  through  innovative  program  structuring. 
These  improvements  would  be  the  result  of  a  better  understanding 
of  overall  considerations  necessary  for  improved  R&M  and  the 
associated  risk  of  management  decisions.  In  addition,  this 
section  provides  guidelines  that  any  program  manager  or  chief 
engineer  can  build  on  to  improve  their  own  abilities  for  improved 
design  and  management. 
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A.  VARIATIONS  IN  PROGRAMS  AND  THE  ACQUISITION  ENVIRONMENT 


Variation  in  programs  and  variations  in  the  acquisition 
environment  were  considered  in  reviewing  programs.  Difference 
in  the  types  of  systems/  the  expected  operational  environment 
and  usage,  the  technology  aspects  of  the  system  as  well  as  the 
acquisition  environment  existing  at  the  timef  played  major  roles 
in  associated  program  structuring. 

1.  Variations  in  Programs 

An  analysis  across  the  F-15  (APG-63),  F-16  (APG-66)  the 
F/A-18  (APG-65),  T700  and  other  case  studies  as  well  as  other 
program  reports  shows  that  the  methods  used  by  the  Services  to 
structure  programs  must  consider!  what  type  of  system  is  being 
procured/  what  operational  environment  and  usage  is  intended, 
and  what  technology  applications  might  be  used.  To  structure 
programs,  review  and/or  analyze  programs  and  manage  programs, 
careful  consideration  must  be  given  to  these  areas  as  there  is 
no  single  rule  of  thumb  which  addresses  all  concerns  in  struc¬ 
turing  proarams  for  R&M . 

a.  Type  of  System.  R&M  elements  needed  in  programs  are 
dependent  on  and  should  be  tailored  to  the  type  of  system. 
Considerations  for  mechanical  systems  such  as  engines  might  be 
similar  to  electronic  systems  in  some  areas  and  grossly  different 
in  others.  Requirements  for  a  "thermal  analysis,"  for  example, 
would  be  a  similar  program  element  for  both  engines  and  radars. 

A  requirement  for  "derating  criteria,"  which  is  important  in 
electronics  parts,  would  have  little  meaning  to  an  engine  contractor 
but  "margin  of  safety"  or  allowable  strengths  of  material  would 
have  considerable  meaning.  Similarly,  other  considerations  may 
came  into  play  in  other  technical  areas  such  as  electrooptical . 

The  structuring  presented  in  this  volume  was  derived  from  reviews 
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of  high-performance  radars,  and,  while  it  might  be  representative 
of  a  class  of  complex  electronics,  it  does  not  necessarily 
accommodate  all  types  of  systems,  although  some  similarities  may 
exist . 


b.  Operational  Environment.  The  installation  and  usage  of 
hardware  dramatically  influences  the  resulting  reliability  of 
the  hardware.  As  an  example,  the  Magnavox  ARC-164  UHF  Radio, 
widely  used  in  the  Air  Force  inventory,  has  dramatically  different 
removal  experience  depending  on  the  aircraft  installation  involved 
as  shown  in  Fig.  B-l. 


FIGURE  B-l.  Impact  of  Installed-Use  Environment 
(Magnavox  ARC-164  UHF  Radio,  AFM  66-1  Data,  Jan-Dec  1982) 
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The  UHF  radio  and  TACAN  experience  points  out  that  there 
is  inherent  uncertainty  in  attempting  to  translate  field  occur¬ 
rences  to  design  features/attributes  of  one  program  for  use  in 
planning  another  program  without  first  attempting  to  understand 
the  many  subtle,  but  potentially  major,  impactors . 

As  another  example,  in  the  case  of  the  F-15  design,  the  20mm 
Gatling-type  gun  (M61-A1)  is  located  on  the  right  side  of  the 
airplane  adjacent  to  the  wing  root,  above,  outboard,  and  aft  of 
the  right  engine  air  inlet  plane.  Because  of  the  remote  location 
of  the  gun,  the  radar  peak  "g"  level  is  only  5  g's  compared  to  24 
g's  for  the  F-4E  (APQ-120)  radar  which  has  the  gun  in  the  nose. 

For  the  F-16,  the  gun  is  located  on  the  left  side  of  the  airplane 
in  the  wing  root  (strake)  area,  aft  and  above  the  engine  air 
inlet  so  that  it  too  has  a  minimum  effect  on  the  radar.  On  the 
F/A-18,  the  gun  is  located  in  the  nose  above  the  radar.  Under¬ 
standing  the  potentially  adverse  effects  of  this  configuration 
allowed  the  effects  to  be  minimized  by  providing  sealing  from 
the  gun  gas  and  vibration  isolators  at  the  radar  interface  between 
the  rack  and  aircraft  bulkhead. 

Similarly,  there  are  major  variations  which  might  occur 
when  going  between  different  relative  military  user  environments 
(e.g..  Navy — salt  and  sea;  Army — mud  and  sand)  and  resource 
environments  (e.g.,  skills,  facilities,  etc.).  Thus,  in  reviewing 
or  planning  programs,  R&M  elements,  schedules  and  resources  must 
consider  these  types  of  factors. 


c.  Technology  Applied.  The  evolutionary  stage  of  the 
hardware/software  involved  and  the  degree  of  the  state-of-the-art 
design  in  a  system  play  a  major  role  in  attempting  to  structure 
R&M  elements  into  a  program.  While  a  fifth-generation  system  of 
an  evolutionary  technology  might  effectively  incorporate  changes 
to  improve  R&M  as  a  result  of  test  and  field  experience,  a  brand 
new  first-generation  state-of-the-art  system  would  in  all 
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probability  require  special  considerations  which  are  sufficiently 
flexible  to  fit  within  the  design,  test-analyze-and-f ix  processes 
associated  with  meeting  functional  performance  requirements  with 
the  new  design  technology. 

In  the  case  of  the  radars  studied,  the  basic  radars  were 
developed  through  an  evolutionary  process?  evolution  of  the 
P-15  Radar,  after  original  design,  is  shown  in  Fig.  B-3. 

A  number  of  electronic  technologies  became  available  in  the 
late  1960s  that  allowed  the  formulation  of  a  radar  system  that 
provided  detection  and  tracking  over  all  aspects  of  look-down  in 
ground  clutter.  This  radar,  the  APG-63,  is  a  high-power,  coherent 
pulse  doppler  system  that  combined  the  proven  long-range  head-on 
and  look-down  detection  capability  of  a  high  PRF  system  with  a 
new  Airborne  Moving  Target  Indication  ( AMTI )  waveform  in  the 
medium  PRF  region  for  shorter-range ,  all-aspect  look -down 
capability  necessary  for  air  combat  tactics.  The  design 
flexibility  of  diverse  waveforms,  high  and  medium  PRFs,  were 
made  available  through  the  development  of  the  shadow  grid 
traveling  wave  tube  transmitters.  The  low  antenna  side lobe 
levels  that  are  critical  to  the  new  medium  PRF  AMTI  mechanization 
were  made  possible  by  precise  control  of  planar  array  radiation 
distributions,  another  new  technology  development. 

The  development  of  all  solid-state  digital  signal  processing 
was  the  most  significant  reliability-related  event  in  the  APG-63 
radar  evolution,  literally  revolutionizing  radar  design  concepts 
by  allowing  a  mechanization  which  shifted  hardware  functions  to 
software  functions  and  thus  greatly  improving  the  reliability. 

A  newly-developed  digital  multiplex  bus  simplified  the  radar  in¬ 
terface  with  other  avionics  systems  and  provided  the  communica¬ 
tion  link  necessary  for  on-board,  Built-In-Test  (BIT)  and  System 
Test  (ST),  thus  improving  both  reliability  and  maintainability 
(diagnostic).  The  continuing  progress  in  digital  technology  has 
allowed  the  radar  to  grow  both  in  performance  and  reliability. 

826/1-7 

B-8 


Radar  System  Evolution 


A  Programmable  Signal  Processor  (PSP)  was  introduced  into 
the  APG-63  in  1980,  and,  besides  changing  previously  hard-wired 
digital  processing  functions  to  software,  it  provided  higher 
reliability  through  the  use  of  higher  density  4K  BIT  RAM  devices. 

A  16K-word  ferrite-core  memory  general-purpose  digital 
computer  was  initially  used  in  the  APG-63  to  provide  automatic 
control  of  all  radar  modes.  This  included  all  Built-In-Test 
routines  and  BIT  test  result  storage  in  a  coded  matrix  that  is 
made  available  for  playback  for  the  maintenance  crew.  A  break¬ 
through  in  solid-state  Electrically  Alterable  Read-Only  Memory 
(EAROM)  devices  allowed  the  introduction  of  a  24K-word  Solid- 
State  Memory  Computer  in  1977  as  a  value  change  program  cost 
savings.  This  computer  had  the  growth  capability  to  96K-word 
(or  96K  x  24  Bit  =2.3  Megabit)  capacity  by  module  addition.  A 
portion  of  this  was  used  to  extend  the  Built-In-Test  capability. 

In  1980,  the  increased  storage  was  used  in  conjunction  with  the 
PSP,  which  resulted  in  improved  fault  detection/isolation  in 
some  LRUs . 

The  continuous  application  of  later  technology  has  allowed 
the  radar  to  achieve  reliability  and  maintainability  growth  and 
to  reach  the  present  part  count  of  18,800,  down  some  4,000  parts 
from  the  complexity  which  existed  for  the  R&D  program.  Improved 
F-15  C/D  radar  R&M  is  evident  in  the  AFM  66-1  reports  compared  to 
the  earlier  F-15  A/B  radar  experience.  This  is  largely  attributed 
to  the  May  1980  introduction  of  the  Programmable  Signal  Processor 
and  the  improved  solid-state  digital  memory  computer  (Ref.  F-15 
Case  Study,  pp.  IB-2,-3). 

Similarly,  the  F-16  Radar  was  the  result  of  design  evolution 
as  shown  in  Fig.  B-4.  This  evolution  led  to  the  development 
of  the  modular  APG-66  radar  used  on  the  F-16.  Westinghouse 
began  design  and  development  activity  in  1971  for  a  new  series  of 
modular  radars.  The  WX  series  of  radars — and  in  particular  the 
WX-200--used  the  pulse  doppler  principle  and  advanced  digital 


826/1-9 


m 


U  O'  V 

3  — 
O 

O  O  ^ 
<r  k  u  : 
«o  a  u  c 
o%  x 
-  •  <  3 
sO  >  *■ 
«J  l  O  - 
-J  O  <  3 
<  & 

If  <  • 


at  k 

H  < 
Ilk  JV 
x  fit  o  *t 
0.  >  o 

u  J  i  Z 

o  Z  Ui 

iSi* 

J3ki 

<  O  J  at 

i  3  o  •  o 

>  O  X  K\  03 


x5  2 
auz  h yy^ 

<<ua.h  v 

ui  at  u  <  ui  r 


—  U) 

i  3 

u 

X 

u  o  at 

<  —  t- 

«  <  u 

Ml 

<  I  I 


>-  O  Q  l/l  <  H- 
kttllUBhX 
—  O  U  O 

Ji5  5  •°'- 

iuzo«-x 

>  ttl  <  IA  U.  l» 

1  i  •  It 


<5  8 

Q  H  UI  2 

a.  o  a  x 
szguu 
jo 
S3  <a  ui 
^  at  h  < 

<*  x  x  x  o 

Hdfu 
*>  —  <  > 
■*  W  J  0£  o 

*  <  o  o  a. 
oat  a.  ac  x 
o  k  »«  a  - 


/ 


~  3 

at 

> 

J  -  (rt  fl. 

o- 

mi 

-jo  x 

V) 

mi 

<  <  at  o 

V) 

< 

u  <  u 

UI 

at  u 

2  X  o  ^ 

2  U  <  O 

u 

H  — 

o 

X  at 

O  z  at  o 

at 

X  H 

at uj 

a.' 

u 

H  >  X 
UtOO 

?j 

at  uj 

UI  mt 

UJ  K\  <  «-< 

—  < 

2“ 

-J 

h  x  at 

UJ  •  •  •  v 

U  O  UJ 

a.  h 

J 

3  -  H  X 
Q  UI  3  O 

p.  . 
r  \  I 


oc  j  x  ui  ui 
a.  <o  o.  j 

HU  H 
*o  -  XX 
O  *  O  O 

O  Q  t  I  j 


*-  ac  ► 
<  uj  < 
O  i£  mJ 

n.  *  u  a. 
3  u  <  in 
_  X  u  oc  — 
lUUhfl 
< 

i/i  a  u  o 
U  111  z  w 

O  J  o  —  > 
»-Z  JO 

*  ?  H  £ 

§2f*£ 

J 


UI 

at  u 

<  o 

o  at 

<  a. 
at 

at 

<  o  < 

o  Ui  z 

<  -J  o 

et  JO" 

O  X  o» 

at  at 

<  H  O  UJ 

-J  X  -I 

3010 

O  u  at  < 

«£§1 

oiiao 

o<  —  iu  ac 

2  *  •  •  • 

J 


O 

2  ^ 

•—  K 

</> 
u> 

UJ  *> 


a.  o 
o>  ’“■*» 

a.  <  — 


o» 

I  22 

u  ax 

►-m  <  X  <  —  o 
i- Jtiau  -i-o 
Msaocuiauiu 


‘  '  .Vv’Vn  V-  •*. - 

,.v-%  v.v.' 


v^Vv’*  -  '.  >Nv 


’.•\-W.v 


AD-A142  509  IDfl/OSD< INSTITUTE  FOR  DEFENSE  ANALYSES/OFFICE  OF  THE  2/5 

SECRETARY  OF  DEFENSE  (U)  INSTITUTE  FOR  DEFENSE 
ANALYSES  ALEXANDRIA  VA  J  R  RIVOIRE  ET  AL.  NOV  83 
UNCLASSIFIED  IDA-R-272-VOL-3  IDA/HQ-83-25967  F/G  15/3  NL 


MICROCOPY  RESOLUTION  TEST  CHART 

NATIONAL  BUREAU  Of  STANOAROS-I9M  A 


•BN 

i 


y 


1 


techniques.  Demonstration  of  these  balanced  design  techniques  led 
directly  to  the  subsequent  balanced  design  and  development  of  the 
APG-66  in  July  1974  (Ref.  F-16  Case  Study,  pp.  IB-12). 

The  F/A-18's  APG-65  radar  draws  on  all  of  the  APG-63  experi¬ 
ence  and  contains  a  completely  programmable  signal  processor 
that  is  similar  to  the  one  used  on  the  F-15  C/D  radar  and  has  at 
least  one-third  of  the  circuit  boards  that  are  common.  Figure  B-5 
depicts  the  system  evolution. 

The  F/A-18  APG-65  radar  contains  more  computer  memory  than  any 
other  current  production  fighter.  This  is  due,  in  part,  to  the 
numerous  radar  modes  and  to  the  large  storage  requirements  of  the 
multi-mode  programmable  signal  processor  (Ref.  F/A-18  Case  Study, 
p.  25).  The  following  summarizes  current  memory  capabilities  of 
the  three  airborne  radars  studied: 


F-15 

96K 

24-Bit 

words 

F-16 

48K 

16-Bit 

words 

F/A-18* 

21  OK 

16-Bit 

words 

In  addition  to  the  basic  evolution  ot  hardware  systems,  the 
introduction  of  new  component  technologies  has  impacted  overall 
R&M  and  associated  program  elements  and  structures.  Figure  B-6 
shows  the  relative  influences  of  technology  introduction  in  the 
form  of  the  vacuum  tubes  and  the  subsequent  transition  to  solid- 
state  components  and  then  integrated  circuits/microcircuits. 

For  early  vintage,  predominantly  vacuum  tube  radars  such  as 
the  MG-13  (F-101B)  and  the  APQ-72  (F-4B),  average  part  failure 
rates  on  the  order  of  35  in  1,000,000  hours  are  shown  as  derived 
from  field  reliability  data.  With  the  technology  shift  toward 
solid-state  (transistors  and  diodes)  as  incorporated  in  the 
APQ-120  (f-4E),  parts  failure  rates  improved  to  an  average  part 
failure  rate  on  the  order  of  10  in  1,000,000  hours,  as  shown. 

Now,  with  modern  radar  systems  such  as  the  APG-66  (F-16A),  APG-63 


^Includes  30K  16  BIT  words  dedicated  to  fault  detection  and 
isolation  (Note:  F-14  total  memory  is  only  30K). 
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(P-15  C/D),  and  the  APG-65  (F-18),  which  utilize  integrated 
circuit/microelectronic  devices,  the  average  part  failure  rate 
has  improved  to  the  order  of  2.5  in  1,000,000  hours. 

To  realize  fully  the  benefits  of  the  inherent  improvement 
in  components  and  the  resulting  improvement  in  system  reliability, 
new  programmatic  elements  for  R&M  had  to  be  evolved  with  the 
technology  (e.g. ,  the  introduction  of  junction  temperature  as  a 
new  parameter  for  reliability  design  consideration  for  solid- 
state  devices).  Similarly,  as  new  and  advancing  technologies 
surface  in  the  future,  additional  elements  will  surface  for 
consideration  in  structuring  and  conducting  R&M  programs. 

The  evolutionary  process  seen  in  the  development  of  radars 
can  also  be  illustrated  in  the  T700  engine  development.  A  series 
of  engine  component  advanced  technology  programs  on  compressors, 
turbines,  recuperators,  inlet  particle  separators,  etc.,  were 
initiated  in  anticipation  of  future  aircraft  requirements.  These 
concurrent  component  developments  evolved  into  a  competitive 
demonstration  program  that  validated  the  new  engine  state-of-the- 
art  technology  advancement  and  reduced  the  risk  of  concurrency  in 
the  engine  and  aircraft  development.  The  advanced  technology 
program  illustrated  in  Fig.  B-7  was  a  significant  contribution 
to  the  high  reliability  and  reduced  maintenance  workload  of  the 
T700  engine. 
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ADVANCED  TECHNOLOG  ST— MODERN  ENGINE 


•  LOW  FUEL  CONSUMPTION— OPTIMIZED  FOR  CRUISE 


BUILT  FOR  THE  ENVIRONMENT 


-  INTEGRAL  SAND  SEPARATOR 


-  BUILT-IN  COMPRESSOR  CLEANING 


-  RUGGED  COMPRESSOR 


SIMPLIFIED  PILOT  CONTROL 


•  REDUCED  MAINTENANCE  WORKLOAD 


-  NO  ADJUSTMENTS 

-  10  STANDARD  TOOLS  FOR  ALL  UNIT  AND 

INTERMEDIATE  MAINTENANCE 

-  HIGH  RELIABILITY/LONG  LIFE 


FIGURE  B-7.  T700  Engine  Characteristics 
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2.  Variations  in  the  Acquisition  Environment 


Many  variations  exist  in  the  acquisition  process.  This 
section  provides  a  discussion  of  the  changing  emphasis  that 
the  acquisition  environment  (time)  has  had  on  R&M  requirements, 
the  changes  brought  about  through  new  directives  and  standards, 
funding  characteristics,  and  the  effect  of  the  DoD  acquisition 
policy,  i.e.,  concurrency. 

a.  A  Changing  Emphasis  for  R&M.  The  acquisition  environment 
for  major  weapon  systems  has  varied  considerably  over  time  with 
respect  to  R&M.  Generally,  considerations  for  R&M  are  most 
apparent  during  peacetime  and  tend  to  diminish  during  wartime. 

The  APQ-120  radar  for  the  F-4E  aircraft  is  an  example  of  a  wartime 
(Vietnam)  development  where  equipment  delivery  was  the  most 
important  consideration  and  as  a  result,  R&M  provisions  were 
deferred  until  the  system  was  fielded.  Prior  to  1970,  aircraft 
ground  support  equipment  (GSE)  was  exempted  from  R&M  consideration 
in  the  major  system  development.  As  system  complexity  increased, 
GSE  R&M  had  to  be  expanded  to  preclude  more  problems  with  ground 
equipment  than  with  the  aircraft  it  was  to  support. 

More  recently,  emphasis  on  flight  simulators  highlights  new 
areas  for  concern.  These  systems,  which  generally  are  more  com¬ 
plex  and  costly  than  the  systems  they  simulate,  have  tradition¬ 
ally  been  developed  under  R&M  programs  patterned  after  the  aircraft 
systems.  Today,  considering  the  difference  that  environment  and 
use  place  on  these  systems,  relative  to  the  systems  they  simu¬ 
late,  R&M  programs,  structured  for  the  unique  requirements  of 
simulators,  are  in  order. 

b.  DoD  Directives/MIL  Standards.  The  character  of  R&M 
programs  has  varied  over  time  as  new  weapon  systems  have  been 
developed.  Requirements  for  obtaining  R&M  objectives,  as  stated 
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in  request  for  proposals,  have  reflected  the  popular  notions  at 
the  time  as  well  as  the  then  current  world  situation.  Several 
documents  have  had  a  significant  effect  on  weapon  system  devel¬ 
opment  and  are  identified  here: 

e  1957 — Advisory  Group  on  Reliability  Electronic  Equipment 
(AGREE)  Report.  Task  3  of  the  AGREE  report  set  forth 
sequential  testing  with  the  concepts  of  Specified  MTBP 
(0q),  Minimum  Acceptable  MTBP  (e^),  Discrimination 
Ratio  ( 8 0/ ® 1 )  and  Decision  Risks  (a,  8). 

Sequential  testing  grew  in  popularity  and  began  to  replace 
fixed  length  testing  as  a  requirement  in  many  contracts. 

e  MIL— STD— 785B,  15  SEPTEMBER  1980,  RELIABILITY  PROGRAM  FOR 
SYSTEMS  AND  EQUIPMENT  DEVELOPMENT  AND  PRODUCTION.  This 
military  standard  consists  of  basic  application  require¬ 
ments,  specific  tailorable  reliability  program  tasks,  and 
an  appendix  which  includes  an  application  matrix  and  guid¬ 
ance  and  rationale  for  task  selection.  Provisions  include 

a.  Emphasis  on  reliability  engineering  tasks  and  tests. 

b.  A  sharp  distinction  between  basic  reliability  and 
mission  reliability;  measures  of  basic  reliability 
such  as  Mean-Time-Between-Failure  ( MTBF )  which  include 
all  item  life  units  (not  just  mission  time)  and  all 
failure  within  the  item  (not  just  mission  critical 
failures  of  the  item  itself);  and  application  of  basic 
reliability  requirements  to  all  the  items. 

e  MIL-STD-470A,  3  JANUARY  1983,  MAINTAINABILITY  PROGRAM  FOR 
SYSTEMS  AND  EQUIPMENT.  This  Military  Standard  consists  of 
basic  application  requirements,  specific  tailorable  main¬ 
tainability  program  tasks,  and  an  appendix  which  includes 
an  application  matrix  and  guidance  and  rationale  for  task 
selection.  Provisions  include: 


826/1-17 


B-18 


a.  Emphasis  on  the  need  for  including  testability  consid¬ 
erations  as  part  of  the  Maintainability  Program. 
Recognition  has  been  given  the  fact  that  Built-In- 
Test  (BIT),  external  test  systems  and  testers  criti¬ 
cally  impact  not  only  the  attainment  of  maintainability 
design  characteristics  but  acquisition  and  life-cycle 
costs  as  well. 

b.  Emphasis  has  been  placed  on  considering  maintainability 
program  needs  at  all  three  levels  of  maintenance 
(organizational,  intermediate,  and  depot.) 


MIL-STD-471A,  10  JANUARY  1975,  MAINTAINABILITY  VERIFI- 


CATION/DEKONSTRATION/EVALUATION.  This  standard  defines 


a  planned  program  for  verification,  demonstration  and 
evaluation  of  the  achievement  of  specified  maintain¬ 
ability  requirements  and  for  the  assessment  of  impact 
on  planned  logistic  support. 


e  1965 — MIL-STD— 781A — Reliability  Tests;  Exponential 
Distribution.  This  military  standard  reflects  the 
concepts  of  the  AGREE  report  and  expands  the  applica¬ 
tion  to  include  a  "shopping  list"  of  test  plans  and 
test  levels.  A  method,  which  was  later  dropped  in  the 
"B"  and  subsequent  revisions,  for  determining  the  num¬ 
ber  of  production  test  articles  based  on  MTBF,  test  time, 
and  production  rate  was  included.  Plan  III  and  Level  F 
which  specified  temperature  variation  from  -54°C  to  +71°C 
and  sine  vibration  of  2.2  g's  peak  acceleration  at  any 
nonresonant  frequency  between  20  and  60  cps  was  typical. 
Revision  C,  dated  21  October  1977,  is  now  in  effect  on 
newer  programs,  and  replaces  the  vibration  requirement 
with  random  vibration  and/or  swept  sine  vibration  from  20 
to  2000  cycles,  a  more  realistic  representation  of  modern 
jet  aircraft  vibration  environments. 


826/1-18 


B-19 


•  1977  NAVMATINST  3000. 1A.  This  instruction  formalized 
a  shift  in  the  Navy  approach  to  R&M  from  numbers/test 
approach  to  the  design/manufacturing  approach  that  was 
used  successfully  at  NASA. 

•  1980  DoD  Directive  5000.40.  This  directive  establishes 
policies  and  responsibilities  for  the  reliability  and 
maintainability  of  systems,  subsystems  and  equipment. 

The  directive  set  forth  the  policy  of  defining  funda¬ 
mentals  of  design,  manufacture  and  management  which 
would  result  in  delivery  of  reliable  and  maintainable 
hardware  systems  to  the  operational  forces.  It  addresses 
the  R&M  achievements  that  shall  be  accomplished  during 
all  phases  of  the  acquisition  process  (i.e,  conceptual 
demonstration/validation  FSD,  production  and  deployment 
phases  and  during  in-service  evaluation).  It  addresses 
most  of  the  front-end  R&M  design  elements  (stress  analyses, 
FMEA,  thermal  analysis,  derating,  etc.),  R&M  growth  testing 
ESS  and  Failure  Reporting,  Analysis  and  Corrective  Action 
System  (FRACAS)  and  R&M  Accounting  Policy. 


c.  Funding  Characteristics.  The  Services  procure  weapon 
systems  with  somewhat  different  R&M  philosophies.  Meeting  the  R&M 
requirements  in  a  cost-effective  manner,  however,  is  a  common 
problem  for  all  programs.  Structuring  a  R&M  program  for  a  weapon 
system  that  is  being  developed  under  severe  budget  constraints 
requires  many  trade-offs  to  obtain  the  desired  balance  between 
performance  and  R&M  requirements.  These  trade-offs  must  be  done 
early  in  the  program  to  identify  the  elements  that  will  have  the 
highest  payoff  in  enhancing  the  R&M  of  the  system.  To  the  extent 
performance  requirements  will  permit,  introduction  of  immature 
technology  should  be  minimized.  In  other  words,  the  use  of  proven 
technology  and  existing  design  where  possible  will  minimize  down¬ 
stream  problems. 

Today's  funding  profiles,  as  shown  on  Fig.  B-8,  tend  to  drive 
technical  decisions  which  can  lead  to  poor  initial  reliability. 


FUNDING 

RAT* 


FIGURE  B-8.  Funding  Profiles 
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The  proposed  profile  is  intended  to  reflect  the  need  for 
early  program  funding.  During  these  early  phases,  high-risk 
portions  of  the  program  should  be  identified  and  actions  taken  to 
minimize  those  risks. 

Note  that  a  significant  amount  of  development  funding  is 
required  prior  to  Milestone  II,  as  well  as  some  funding  after 
Milestone  III.  This  post-Milestone  III  funding  will  permit  the 
engineering  effort  necessary  to  correct  problems  found  after 
production  begins. 

The  general  shape  of  the  actual  Total  Obligation  Authority 
(TOA)  profile  for  the  FSD  portion  of  the  F-15,  F-16  and  F-18 
programs  is  shown  in  Figures  B-9,  B-10  and  B-ll,  respectively. 
Milestone  events  have  been  added  to  the  figures  to  show  how 
these  programs  were  funded  relative  to  the  milestones.  A  com¬ 
parison  with  the  recommended  profile  shown  in  Figure  B-8  tends 
to  reflect  the  amount  of  concurrency  in  each  program  as  well  as 
how  these  programs  were  funded  relative  to  the  proposed  funding 
profile . 

For  programs  with  severe  budget  constraints,  consideration 
must  be  given  to  focusing  on  front-end  design  activities  to  develop 
high  design  potential  and  then  planning  for  an  extended  R&M  growth/ 
maturation  program,  using  data  obtained  from  production  and  field 
usage  to  identify  problems  and  establish  design  fixes.  Costs  of 
R&M  testing  (R&M  Development  and  Demonstration)  can  be  minimized 
by  using  data  from  all  tests  for  R&M  purposes  and  combining  R&M 
test  requirements  with  performance  and  environmental  qualification 
tests.  If  results  are  to  be  achieved  which  are  considerably  better 
than  those  currently  being  experienced,  a  need  exists  to  do  some 
things  differently.  Starting  a  growth  program  with  a  more  mature 
design  and  growing  the  diagnostics,  simultaneously,  is  a  promising 
approach  favored  by  many  experts  on  this  study. 

In  the  past  it  was  not  uncommon  to  begin  growth  testing  with 
an  immature  design.  Test  time  was  wasted  identifying  the  major 
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FIGURE  B-9 .  F-15  Development  Cost  Profile 
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FIGURE  B-ll .  F/A-18  Development  Cost  Profile 
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faults  that  could  have  been  eliminated  with  a  stronger  up-front 
effort.  This  up-front  effort  would  facilitate  more  efficient 
use  of  test  resources  to  identify  the  latent  problems  that  design 
analysis  will  not  detect.  The  funding  profile  proposed,  as  shown 
in  Fig.  B-8,  could  facilitate  this  front-end  effort.  With  the 
present  funding  profile,  the  designer's  innovative  and  creative 
thinking  can  be  severely  constrained  by  schedule.  His  primary 
emphasis  is  to  put  a  system  together  that  functions  with  a  minimum 
of  effort  and  to  develop  a  design  which  not  only  functions  but 
also  does  so  with  a  low  failure  rate.  This  constraint  currently 
forces  the  designer  or  program  manager  to  delay  the  development 
of  a  diagnostic  system  until  well  into  the  testing  phase,  a 
major  problem  identified  by  these  case  studies. 

In  order  to  make  a  major  breakthrough  in  reliable  systems, 
the  funding  profile  proposed  or  one  similar  to  it  is  probably 
mandatory.  Not  only  does  this  make  the  growth  and  maturation 
phase  workable  but  it  also  reduces  the  burden  of  test  concurrency 
and  would  allow  the  growth  programs  to  start  with  a  system  which, 
by  current  standards,  would  be  equivalent  to  a  second  test  phase 
system. 

d.  Acquisition  Policy.  The  government  acquisition  policy 
must  be  considered  when  structuring  a  program.  When  the  policy 
was  one  of  "fly  before  buy"  there  was  adequate  opportunity  to 
develop,  test  and  fix  problems  before  committing  to  production. 

The  T700  engine  program  was  a  sequential  type  development  but 
was  concurrent  with  the  Blackhawk  and  Apache  programs  (Ref. 

T700  Case  Study,  p.  IC-15).  The  design  and  development  of  the 
T700  engine  started  with  early  component  development  work  in 
1964  and  culminated  with  shipment  of  the  first  production  engine 
in  March  of  1978,  as  illustrated  in  Fig.  B-12.  The  success  of 
the  engine  is  attributed  to  early  anticipation  of  future  require¬ 
ments  in  the  component  development  efforts,  validation  of  the 
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technical  requirement  in  a  competitive  engine  demonstration  program 
and  concurrent  engine  and  aircraft  full-scale  development  efforts, 
as  illustrated  in  Fig.  B-13.  A  major  contributor  to  the  overall 
program  success  was  adequate  funding  to  complete  satisfactorily 
each  phase  of  the  12-year  development  process.  Concurrent  programs, 
such  as  the  F-15,  F-16,  and  F/A-18,  must  be  structured  so  that  R&M 
activities  are  implemented  at  the  beginning  of  the  program  and  are 
considered  as  important  as  performance  requirements  throughout  the 
development  cycle.  As  discussed  in  Section  D  of  this  appendix, 

R&M  requirements  can  be  met  in  a  concurrent  type  program  if  the  R&M 
design  diciplines  are  enforced  up-front  and  a  R&M  growth  program  is 
implemented  extending  throughout  the  FSD  phase  and  into  production. 
All  R&M  problems  will  not  be  identified  prior  to  initial  deployment 
but  with  a  good  R&M  growth  program  the  R&M  field  requirements  can 
be  met  as  early  or  earlier  than  in  a  sequential  program.  The  F-15, 
F-16,  and  F/A-18,  TPQ-36  and  TPQ-37  were  successful  in  concurrent 
development/production  type  programs. 
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B.  STRUCTURING  FOR  R&M  IN  WEAPON  SYSTEM  PROGRAMS 


This  section  discusses  the  structuring  and  execution  of 
reliability  and  maintainability  programs  as  indicated  by  the 
results  of  the  case  study  reviews  and  other  information  that  was 
presented  or  obtained  during  the  course  of  the  study.  To  aid  in 
the  conduct  of  the  case  study  reviews,  the  R&M  program  activities 
were  divided  into  five  major  categories:  contracting,  management, 
design,  manufacturing,  and  test  and  evaluation.  These  five  major 
categories  were  used  to  provide  the  format  for  the  following 
discussion  on  how  to  structure  and  execute  successful  reliability 
and  maintainability  programs. 

A  major  problem  confronting  procuring  activities  responsible 
for  R&M  is  the  proper  selection  of  key  elements  necessary  for 
successful  programs.  This  is  not  a  simple  task.  Relative 
importance  and  applicability  of  R&M  program  elements  will  vary  to 
some  degree  from  program  to  program  and  phase  to  phase.  Never¬ 
theless,  the  following  paragraphs  provide  some  guidance  and 
rationale  to  assess  whether  or  not  key  program  elements  have 
been  properly  addressed.  Those  R&M  program  elements  discussed 
in  the  following  paragraphs  were  selected  because  of  their 
importance  and  broad  applicability. 


Contract inc 


Seven  elements  will  be  examined  in  this  section.  These 
are:  R&M  requirements,  mission  profile  establishment,  life 
profile  establishment,  R&M  failure  definition,  incentives,  source 
selection  criteria,  and  life-cycle  cost  consideration. 


a.  R&M  Requirements.  Acquisition  programs  must  be  structured 
so  that  full-scale  development  and  production  contracts  contain 
R&M  requirements,  not  goals,  in  order  to  allow  R&M  to  compete 
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effectively  for  management  support.  To  be  truly  enforceable, 
requirements  must  have  contractually  specified  verification. 

R&M  verification  has  historically  been  a  problem  area.  It 
requires  adequate  definition  of  the  quantity  to  be  measured,  the 
method  of  measurement,  and  the  environment  in  which  the  measurement 
is  made.  Divergence  of  any  of  these  factors  from  actual  field 
application  can  result  in  significant  differences  between  verifi¬ 
cation  results  and  results  in  actual  operational  use. 

(1) .  R&M  Contract  Goals  versus  Requirements.  R&M  con¬ 
tract  goals  do  not  receive  proper  attention.  When  R&M  values  were 
specified  as  goals,  as  was  the  frequent  procedure  a  number  of 
years  ago,  the  technical  performance  requirements  received 
higher  priority  in  the  decisions  of  management.  The  use  of  R&M 
goals  placed  R&M  at  a  disadvantage  with  other  system  attributes 
that  were  specified  as  contract  requirements.  The  result  in 

many  cases  was  lower  than  desired  levels  of  R&M.  To  properly 
compete  for  contractor  management  attention  and  resources,  the 
R&M  numerics  should  be  specified  in  the  contract  (specification 
or  statement  of  work)  as  contract  requirements  as  opposed  to 
goals.  Qualitative  requirements  should  also  be  specified 
(particularly  in  maintainability).  All  contract  requirements 
must  have  contractually  specified  verification  to  be  truly 
enforceable  requirements.  The  requirements  must  address  (or 
recognize)  any  R&M  growth  that  is  planned. 

(2) .  Developing  R&M  Requirements.  Properly  defined, 
realistic  R&M  requirements  are  fundamental  to  successful  R&M 
development.  Program  R&M  requirements  establish  the  basis  for 
determining  the  resources  assigned  to  the  task  and  the  R&M  approach 
during  design,  development  and  production. 
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(3).  Testability  Requirements.  The  establishment  of 
testability  requirements,  particularly  in  the  area  of  built-in- 
test,  has  evolved  slowly  over  the  past  ten  years.  Most  programs 
have  followed  the  lead  of  their  predecessors  and  have  added 
little  to  the  general  knowledge  of  the  procedure  for  stating 
requirements.  Considerations  that  have  been  suggested  to  improve 
the  statement  of  requirements  for  diagnostics  include: 

e  BIT  requirements  should  be  stated  in  terms  of  system 
functions,  i.e.,  can  functions  that  are  not  performing 
properly  be  detected  by  the  built-in-test  system?  This 
statement  would  replace  requirements  that  address  the 
detection  of  all  part  failures. 

•  BIT  requirements  should  account  for  the  criticality  of 
the  function  or  equipment.  There  is  no  logic  to  stating 
the  same  level  of  BIT  for  all  avionics  equipments  in  a 
weapon  system  without  regard  for  the  relative  item  criti¬ 
cality. 

•  BIT  requirements  should  consider  the  type  of  system  or 
equipment  being  addressed.  The  mechanization  of  BIT 
for  a  digital  computer  versus  a  video  display  is  consi¬ 
derably  different.  The  requirements  should  recognize 
this  fact. 

•  BIT  has  two  major  functions — detection  of  a  fault  and 
isolation  of  the  fault.  These  two  functions  are  aimed  at 
two  different  users.  The  operator  or  crew  member  needs 
to  know  that  a  failure  exists.  The  maintainer  needs  to 
know  where  the  failure  is  so  it  can  be  repaired.  These 
two  different  aspects  must  be  considered  in  the  BIT 
requirements. 
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•  Most  BIT  systems  have  monitoring  that  is  done  con¬ 
tinuously  and  monitoring  that  is  done  only  upon  com¬ 
mand  from  the  crew  or  the  maintainer.  These  aspects 
should  have  different  requirements.  This  specific 
definition  of  requirements  aids  in  focusing  the 
designer's  action  to  areas  of  criticality  and  need. 

b.  Mission  Profiles.  Mission  profile  development  should 
begin  in  the  earliest  program  phase  and  continue  to  some  degree 
throughout  the  program.  Detailed  design  analysis  and  decisions 
must  be  performed  in  the  context  of  the  expected  system  environ¬ 
mental  conditions.  If  mission  profile  or  profiles  are  not  defined 
early  in  the  program,  then  critical  design  decisions  may  not 
consider  the  expected  operational  stresses.  Reliability  and 
maintainability  are  functions  of  these  operational  stresses. 
Consequently,  an  accurate  and  relatively  comprehensive  definition 
of  the  anticipated  use  environment  is  necessary  for  the  designer 
to  have  a  reasonable  chance  to  design  equipment  which  will  satisfy 
the  operational  need. 

Mission  profile  definition  should  include: 

-  System  utilization 

-  Maintenance  concept 

-  Time-phase  sequence  of  system  operation 

-  Time-phase  sequence  and  period  of  environ¬ 
mental  conditions. 

Profiles  should  be  defined  using  the  best  available  data, 
but  it  must  be  recognized  that  predicting  system  usage  and 
environments  is  not  an  exact  science.  As  the  system  is  developed, 
continued  effort  must  be  expended  to  substantiate  these  projections 
and  make  any  appropriate  modifications.  Systems  with  significant 
advances  in  capability  over  existing  systems  can  result  in 
unexpected  mission  profiles. 
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FIGURE  B-14 .  Non-Mission  Portions  of  a  System  Life 

Profile  are  Significant  in  Stress  Levels 
and  Duration 
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d.  R&M  Failure  Definition.  The  R&M  requirements  only  have 
complete  meaning  if  established  in  a  defined  context.  Variations 
of  failure  definitions,  time,  critical  failures,  etc.,  result  in 
wide  variations  of  meaning  of  R&M  parameters  and  inadequate 
communication  between  the  government  and  the  contractors. 

The  R&M  requirements  must  be  established  in  a  defined  context. 
They  consist  of  the  following  elements: 

•  Quantified  Parameter 

•  Parameter  failure  definition/scoring  criteria 

•  Expected  operational/f ield  service  conditions 

•  Definition  of  verification  procedures. 

These  elements  must  be  recognized  as  changes  in  R&M  requirements. 

The  definition  of  failure,  man-hours,  maintenance,  operating 
time,  critical  failures,  etc.,  must  be  completed  before  the 
quantitative  R&M  requirements  are  established.  General  defini¬ 
tion  must  be  made  prior  to  detailed  design  activity  if  the  R&M 
requirements  are  to  have  any  impact  on  the  design. 

Contractual  agreements  for  failure  definition,  etc.,  must  be 
made  which  are  mutually  acceptable  to  customer  and  contractor. 

The  definitions  must  be  in  writing  and  established  at  an  appropriate 
point  in  the  contract.  Not  all  can  be  made  at  the  front  end  of 
the  contract. 

Different  definitions  should  be  used  to  define  operationally 
relevant  conditions  and  to  define  contract  chargeable  conditions. 

The  priority  should  be  to  define  these  two  sets  accurately  for 
their  two  distinct  purposes  rather  than  compromising  the  separate 
accurate  definitions  for  a  single  less  accurate  set.  However, 
contract  requirements  must  clearly  support  operational  requirements. 
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e.  Incentives .  R&M  contract  incentives  can  aid  in  developing 
reliable  and  maintainable  systems.  Properly  structured  R&M  contract 
incentives  can  be  used  to  focus  contractor  management  attention  on 
critical  R&M  factors.  R&M  contract  incentives  are  not  substitutes 
for  an  overall  R&M  program,  but  rather  contribute  to  a  good  program. 
The  program  must  contain  clear  requirements  and  proper  funding  to 
accomplish  required  changes  to  the  equipment.  The  R&M  incentive 
approach  to  a  particular  program  should  be  based  upon  the  critical 
R&M  parameters  or  characteristics  for  that  system.  Once  the  criti¬ 
cal  parameters  have  been  identified  the  various  incentive  approaches 
must  then  be  evaluated.  Some  approaches  have  their  basis  for 
measurement  only  in  operational  service  (e.g.,  reliability  improve¬ 
ment  warranties).  Other  incentive  techniques  may  be  applied  in 

any  time  phase  (e.g.,  award  fee).  Careful  attention  should  be 
paid  to  the  selection  and  balance  of  multiple  incentives  to  assure 
that  they  are  compatible,  and  properly  focused  to  achieve  the 
desired  result.  Incentive  applications  should  not  be  limited  only 
to  the  design  phase.  The  R&M  incentives  must  be  viewed  in  the 
context  of  the  total  incentive  package  on  the  contract  to  assure 
proper  relative  weight  is  given  to  R&M  and  that  conflicts  with 
other  incentives  do  not  exist. 

f.  Source  Selection  Criteria.  The  priorities  used  by  the 
government  in  evaluating  potential  sources  for  products  and  awarding 
contracts  have  not  substantiated  the  importance  of  R&M.  The 
government  source  selection  criteria  has  not  consistently  placed 
adequate  priority  on  R&M  (Fig.  B-15). 
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TOTAL  NUMBER  NUMBER  WITH 

OF  PROGRAMS  R&M  IN  SOURCE 

REVIEWED  SELECTION  PERCENT 

ARMY  6  2  33% 

NAVY  6  0  0% 

AIR  FORCE  _8_  _4_  50% 

TOTAL  20  6  30% 

( LMI  Working  Note:  "A  Documentation  of  DoD  Strategies  for 
Acquiring  Weapon  System  Reliability  and  Support"  Dec.  1982, 
pp.  2-2,  2-3,  and  2-4.) 

FIGURE  B-15.  R&M  is  not  Consistently  Used  in  Source  Selection 


The  government's  priority  on  R&M  is  reflected  in: 

-  The  RFP  statement  of  proposal  evaluation  criteria. 

-  The  source  selection  evaluation  board  weighting  method. 

-  The  final  selection  of  a  contractor. 

The  first  of  these  can  motivate  the  contractor  to  propose  his 
best  design  or  approach  to  the  R&M.  The  latter  two  ensure  the 
proper  evaluation  of  the  alternative  R&M  characteristics  and  the 
potential  for  selecting  the  best  approach  from  an  R&M  standpoint. 

Contractors  will  not  attach  high  value  to  R&M  until  they 
see  the  government  making  selection  of  designs  with  good  R&M 
features  and  rejecting  ones  that  trade  off  R&M  design  features. 
R&M  should  be  a  major  element  in  the  evaluation. 


g.  Life-Cycle  Cost  Consideration.  Life-cycle  cost  is  one 
consideration  in  defining  R&M  requirements.  A  system's  life-cycle 
cost  is  a  function  of  R&M  variables  and  can  be  reduced  by  selecting 
and  achieving  certain  R&M  requirements.  Life-cycle  cost  analyses 
can  be  used  to  determine  R&M  levels  that  will  result  in  reduced 
life-cycle  cost.  During  design  trade-off  studies,  LCC  should  be 
used  as  one  factor  to  evaluate  alternative  designs.  Life-cycle 
cost  analyses  must  adequately  consider  other  constraints  such  as 
readiness  requirements  and  overall  system  effectiveness.  During 
test  and  evaluation,  R&M  problems  should  be  assessed  against  LCC 
impact. 
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2.  Management 

In  the  management  area,  three  elements  were  examined.  These 
were:  planning,  control  and  emphasis  by  government  and  contractor 
management,  and  the  monitoring  and  control  of  subcontractors  and 
suppliers.  The  need  for  management  support  of  an  efficient 
engineering  change  system  is  also  discussed. 

a.  Planning,  Control  and  Emphasis.  Government  and  contractor 
top  management  emphasis  can  result  in  enhanced  system  R&M. 

Emphasis  on  R&M  by  government  and  contractor  top  management  can 
provide  a  better  balance  between  R&M  and  conflicting  requirements 
such  as  cost,  schedule,  and  technical  performance. 

Development  programs  contain  many  competing  requirements 
for  resources  and  management  attention.  Management  emphasis  on 
R&M  must  be  evident  to  all  levels  of  government,  contractor,  and 
subcontractor  personnel  to  cause  a  balanced  approach  and  appro¬ 
priate  consideration  to  R&M.  Contractor  management  must  be 
convinced  that  the  government  program  manager  considers  R&M  as 
important  as  other  aspects,  e.g.,  performance,  cost  and  schedule. 
Most  companies  have  the  design  tools  and  systems  and  will  respond 
to  R&M  requirements  if  the  priority  on  this  area  is  evident. 
Government  emphasis  can  be  evidenced  by  source  selection  criteria, 
contract  structure,  incentives,  willingness  to  fund  up-front  money 
for  R&M,  willingness  to  give  R&M  proper  consideration  in  trade-off 
decisions.  Direct  involvement  by  customer  and  contractor  top 
management  is  needed  in  reviews  of  the  design  and  its  R&M  progress. 
Separate  isolated  R&M  reviews  will  not  lead  to  a  balanced  con¬ 
sideration  with  other  contract  requirements. 

b.  Monitoring  and  Control  of  Subcontractors  and  Suppliers. 
Subcontractor  actions  can  be  critical  to  producing  a  reliable 
and  maintainable  system.  On  many  weapon  systems,  most  of  the 
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design,  development  and  manufacturing  is  actually  done  by  subcon- 
contractors.  A  major  aircraft  manufacturer  recently  estimated 
that  only  7  to  10  percent  of  the  weapon  system  unreliability 
was  contributed  by  in-house  design,  while  the  other  90  percent 
to  93  percent  was  contributed  by  subcontractors.  This  makes  it 
critical  that  the  prime  contractor  and  the  government  clearly 
communicate  the  importance  of  R&M  to  the  subcontractors.  The 
prime  contractor  must  carefully  structure  and  manage  subcontracts 
to  assure  that  weapon  system  and  critical  system  R&M  requirements 
are  met. 

The  three  aircraft  radars  examined  were  all  critical  R&M 
items  in  the  prime  contractor's  weapon  system.  The  radars  were 
designed  and  developed  by  subcontractors.  The  radar  subcontracts 
reflected  the  R&M  requirements  and  the  R&M  approach  contained  in 
the  prime  contract.  Where  R&M  contract  incentives  were  included, 
they  were  also  reflected  in  the  major  subcontracts.  The  incen¬ 
tives  and  the  management  priority  placed  on  R&M  by  the  government 
was  understood  and  communicated  to  the  subcontractors  by  the 
prime  contractor  and  the  government. 

All  programs  required  vendors  to  provide  failure  analysis 
and  corrective  action  feedback  on  selected  critical  items. 

c.  Engineering  Change  Process.  An  expedited  change  system 
to  rapidly  approve  and  fund  design  changes  for  R&M  would  give 
the  contractor  tangible  evidence  of  the  government's  interest  in 
R&M.  Valid  and  effective  design  changes  for  R&M  often  take  one 
to  two  years  to  be  approved  and  implemented  after  the  production 
begins.  Some  never  get  funded.  This  indicates  a  low  priority 
on  improving  system  R&M.  The  ability  to  grow  the  R&M  of  the 
production  system  clearly  depends  upon  the  ability  to  make 
engineering  changes.  Many  design  change  systems  are  frustrated 
by  the  government's  long  approval  times  and  funding  constraints. 
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The  government's  reluctance  to  modify  this  process  is  taken  as 
an  indication  of  a  low  priority  on  improving  the  R&M  or  other 
characteristics  of  the  system.  Decisions  to  correct  this 
situation  on  individual  acquisition  programs  would  effectively 
demonstrate  the  government's  priority  on  improving  system  R&M 
through  implementing  design  corrections.  Approval  to  retrofit 
approved  changes  during  the  course  of  contractor  maintenance 
will  also  expedite  the  implementation  of  R&M  improvements. 
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3.  Design 


In  the  very  important  area  of  design,  nine  elements  were 
examined.  These  nine  elements  were: 

-  Development  of  design  requirements 

-  Design  alternative  studies 

-  Design  evaluation  analyses 

-  Parts  and  material  selection  and  control 

-  Derating  criteria 

-  Thermal  and  packaging  criteria 

-  Computer-a ided  design 

-  Testability  analysis 

-  Testability  verification  and  testing. 

a.  Development  of  Design  Requirements.  The  allocation  of 
system  requirements  to  lower  levels  allows  the  designer  and  unit 
managers  to  have  assigned  R&M  responsibilities.  This  allocation 
should  identify  any  areas  of  unusual  technical  risk.  R&M  require¬ 
ments  must  be  translated  into  meaningful  actions  that  the  designer 
can  understand.  In  the  case  of  reliability,  the  designer  cannot 
directly  design  to  requirements  such  as  MTBF.  MTBFs  must  be 
translated  into  the  selection  of  components  and  component  stress 
levels.-  Usually,  this  is  done  by  providing  designers  derating 
criteria,  and  part  selection  guidelines.  Contractors  should 
have  a  rational  translation  process. 

The  process  of  translating  maintainability  requirements  into 
design  actions  is  not  as  well  understood,  particularly  in  the  area 
of  built-in-test.  Nevertheless,  contractors  should  develop  a 
rational  process,  derived  from  top-down  considerations,  to  trans¬ 
late  maintainability  requirements  into  design  guidelines. 

Better  testability  tools  are  needed  to  communicate  the  BIT 
requirements  to  the  designer.  In  the  case  of  reliability,  the 
designer  cannot  respond  directly  to  requirements  such  as  MTBF, 
but  he  can  respond  directly  to  items  such  as  derating  requirements 
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and  parts  selection  guidelines.  Similar  design  criteria  guide¬ 
lines  are  needed  for  the  designer  in  the  area  of  built-in-test. 
Possible  areas  are  items  such  as  components  per  test  point, 
nodes  per  test  point,  frequency  of  examination,  number  of  out-of- 
tolerances  required  prior  to  display  of  indication,  etc. 

Computer-aided  design  tools  exist  to  help  in  providing  rapid 
evaluation  of  the  design  for  testability.  These  tools  aid  in  the 
selection  of  test  points,  evaluation  of  partitioning  and  also 
allow  the  evaluation  of  the  design  and  the  test  points  using 
simulation  of  faults  and  BIT. 

b.  Design  Alternative  Studies.  Trade  studies  have  a  signif 
icant  impact  in  establishing  the  design  baseline  and  in  determini 
the  R&M  of  the  system.  R&M  are  design  attributes  that  must  be 
evaluated  through  analyses.  Design  trade  studies  examine  alter¬ 
native  designs  and  result  in  narrowing  the  range  of  expected  or 
potential  R&M  performance. 

An  aggressively  managed  trade  study  program  is  directed  at 
finding  a  proper  balance  between  the  many  demands  on  the  system 
design.  This  balance  should  attach  appropriate  importance  to 
R&M  considerations  and  risk  to  assure  that  alternatives  which 
improve  R&M  are  examined.  All  trade  studies  need  to  address  the 
impact  on  R&M.  The  studies,  at  the  appropriate  level  of  detail, 
should  be  accomplished  during  all  program  phases. 


c.  Design  Evaluation  Analysis.  To  be  most  effective  the 
R&M  design  evaluation  analyses  must  be  an  integral,  timely  part 
of  the  detailed  design  process.  Otherwise,  they  merely  record 
information  about  the  design  after  the  fact.  The  later  in  the 
process  needed  changes  are  identified,  the  more  costly  in  terms 
of  time  and  schedule  they  become  and  the  lower  the  probability 
that  they  will  be  incorporated. 
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The  design  evaluation  analyses  are  best  done  as  an  integral 
part  of  the  detailed  design,  and  by  members  of  the  design  team 
when  practical.  A  common  flaw  in  some  R&M  programs  is  to  have 
the  R&M  analyses  separated  from  design  activity  by  time  (in  this 
situation  the  analysis  merely  documents  the  characteristics  of 
the  design  without  affecting  the  design  chosen),  distance  or 
organization.  Separation  by  organization  can  have  positive  or 
negative  impact  depending  upon  the  organizational  factors  (and 
personalities)  involved.  The  timing  and  the  credibility  of 
these  analyses  must  be  such  that  they  are  accepted  as  part  of 
the  design  evolution. 

The  role  of  R&M  analyses  in  design  changes  can  often  be  an 
indicator  of  the  importance  placed  upon  these  tasks  by  the  design 
team  and  contractor  management. 

A  technical  "design  point"  should  he  defined  and  specified 
for  the  design  evaluation  analyses.  The  design  evaluation  analyses 
are  inconsistent  (or  inaccurate)  if  varying  environmental  conditions 
are  assumed  for  different  analyses. 

The  design  evaluation  analyses  include  stress  analysis; 
worst  case  analysis;  numerical  predictions;  failure  modes,  effects 
and  criticality  analysis;  and  sneak  circuit  analysis.  Some  of 
these  (e.g.,  predictions,  stress  analysis)  are  dependent  on 
defined  environmental  and  operating  conditions. 

Conditions  selected  for  the  individual  analyses  may  be  different 

-  The  result  is  inconsistent  analyses  on  a  single  * 

development  program 

-  Also,  it  is  difficult  or  impossible  to  do  comparative 
evaluations  between  programs. 


Some  analyses  might  be  done  at  multiple  design  points  (heat 
loads,  operating  modes,  cooling  air  flow  rates,  etc.)  to  completely 
examine  the  likely  system  conditions. 
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d.  Parts  and  Material  Selection  and  Control.  A  well-defined 
properly-executed  parts  (and  material)  control  program  can  reduce 
one  major  source  of  potential  reliability  problems,  piece  part 
failures.  A  parts  control  program  restricts  the  use  of  parts  to 
those  having  well-established  reliability  characteristics  and 
provides  a  methodology  for  managing  exceptions  to  the  control 
program. 

Restricting  appropriate  classes  of  parts  to  we 1 1 -understood , 
high-volume  components  with  increased  assurance  of  part  quality 
can  contribute  significantly  to  producing  a  reliable  product. 

The  approved  parts  list  must  be  available  before  the  designer  is 
ready  to  select  components.  The  authority  for  granting  exceptions 
to  the  control  procedure  should  be  well-defined  and  monitored  to 
assure  that  exceptions  are  not  granted  too  freely. 

e.  Derating  Criteria.  Most  development  programs  impose, 
as  a  requirement,  derating  criteria.  The  derating  criteria 
should  be  part  of  a  derating  program  which  establishes  management 
requirements  and  controls. 

Derating  criteria  should  not  be  blindly  applied  to  all 
components.  Components  have  different  system  criticalities 
and  reliability  sensitivities  to  derating  and  may  require 
different  derating  criteria. 

Exceptions  should  be  justified  on  the  basis  of  cost,  lack 
of  technically  acceptable  substitute,  etc. 

Compliance  with  critical  derating  requirements  should  be 
verified. 

(1).  Derating  Parameters.  Single  parameter  derating 
criteria  can  result  in  inefficient  or  inaccurate  derating.  Some 
development  programs  have  used  simplistic,  one-parameter,  derating 
criteria.  Most  electrical  and  electronic  components  depend  on  a 
combination  of  parameters  to  determine  stress  levels.  For  example 
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Linear  ICs  depend  on 

-  Supply  voltage 

-  Input  voltage 

-  Output  current 

-  Maximum  junction  temperature. 

NPN  transistors  (Fig.  B-16)  depend  on 

-  Case  temperature 

-  Total  power  dissipation. 


Derating  criteria  must  be  based  on  all  essential  parameters 
and  allow  trade-offs  between  the  levels  of  each  parameter.  (See 
RADC  Report  TR  82-177). 


(2).  Exceptions  to  Derating  Criteria.  Exceptions  to 
derating  guidelines  should  be  tightly  controlled.  Unnecessary 
exceptions  or  too  numerous  exceptions  result  in  parts  that  are 
overstressed  or  stressed  beyond  the  intended  limits. 

Most  development  programs  impose  an  acceptable  derating 
criteria.  The  derating  criteria  should  be  part  of  a  derating 
program  which  establishes  management  requirements  and  controls. 
Higher  part  stress  results  in  part  from  lack  of  control  of  the 
exceptions  or  deviations  to  the  derating  criteria. 


f.  Thermal  and  Packaging  Criteria. 


Thermal -packaging  criteria  are  critical  to  the  R&M  success 
of  the  design.  Temperature  level  and  temperature  rate  of  change 
are  the  principal  sources  of  stress  which  affect  component  relia¬ 
bility  (see  Fig.  B-17).  The  thermal  conditions  that  equipment 
parts  are  subjected  to  are  dependent  on  the  operating  environment 
and  packaging/cooling  design.  The  choice  of  cooling  design  can 
significantly  impact  different  component  temperatures  and/or 
rate  of  change  of  temperature.  Cooling  methods  include: 
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FAILURE  RATE 

{FAILURES  PER  MILLION  HOURS) 
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LINEAR  DERATING 


LEVEL 


0.70  (1)1 


PARAMETER 


1  SUPPLY  VOLTAGE 


|  INPUT  VOLTAGE 


|  OUTPUT  CURRENT 

i 

|  MAX  JUNCT.  TEMP 
I  (DEG  C) 


(1)  Dec igning  below  70Z  of  the  supply  voltage  may  operate  the  device 
below  the  recommended  operating  voltage. 


ABSOLUTE  MAXIMUM 


CASE  T|MPERATURE  -  T£ 

Derating  for  NPN  Transistor  Type 
2N3997  By  Junction  Temperature  Only 

FIGURE  B-16.  Multiple  Parameters  for  Derating 

(from  RADC  Report  TR-82-177) 
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-  Direct  cooling; 

-  Cold  walls  or  edge  board  cooling,  and 

-  Flow  through  cold  plate  cooling. 

(See  RADC  Report  TR  82-177). 

After  the  equipment  is  produced,  a  thermal  verification  test 
is  required  to  assure  that  the  thermal  criteria  have  been  met. 

The  test  should  be  run  using  a  simulated  worst-case  mission 
environment. 

Specific  thermal  analysis  and  thermal  analysis  verification 
requirements  should  be  specified  to  assure  compliance  with  thermal 
design  conditions. 

g.  Computer-Aided  Design.  Computer-Aided  Design  (CAD) 
techniques  can  help  produce  reliable  and  maintainable  equipment. 
R&M  characteristics  are  attributes  that  require  evaluation  of 
specific  designs.  CAD  techniques  can  allow  the  designer  to 
quickly  and  efficiently  evaluate  design  alternatives,  allowing 
more  design  alternatives  to  be  evaluated  in  more  detail.  This 
promotes  development  of  a  design  that  satisfies  a  number  of 
design  constraints  and  optimizes  other  design  features. 

Computer  analyses  are  tools  that  in  an  automated  manner 
attempt  to  optimize  a  characteristic  of  the  design,  or  other 
automated  analyses.  CAD  for  R&M  can  consist  of  R&M  predictions 
that  the  designer  can  evaluate,  the  identification  of  over¬ 
stressing,  sneak  paths,  and  malfunction  choke  points,  as  well  as 
automated  decisionmaking  that  forces  the  design  alternatives  to 
meet  certain  constraints.  CAD  techniques  have  been  used  to 
perform  thermal  analysis  (Fig.  B-18).  The  priority  of  R&M  in 
the  CAD  process  should  be  balanced  with  technical  performance, 
weight,  etc.  Viable,  cost-effective  recommendations  should 
result  from  the  process.  Integration  of  R&M  tools  into  the  CAD 
system  will  occur  when  customer  and  contractor  management  are 
motivated  toward  R&M  considerations. 
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FIGURE  B-18.  Computer-aided  design  was  used  to  opti 
board  layout  considering  thermal  stresses 
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h.  Testability  Analysis.  Testability  analysis  should  be 
used  to  define  diagnostic  requirements  before  the  detailed  design. 
During  the  detailed  design,  test  point  analysis  and  nodal  analysis 
for  partitioning  should  be  major  inputs  to  the  layout/packaging 
design.  Analyses  of  projected  BIT  performance  and  fault  simu¬ 
lation  studies  should  be  used  to  evaluate  the  progress  toward 

the  BIT  objectives  and  as  inputs  to  a  BIT  maturation  program. 

Testability  analysis  needs  to  relate  to  the  actual  conditions 
that  will  occur.  Recent  technical  papers  have  highlighted  the 
difference  between  "hard"  failures  that  are  "permanent"  and 
faults  that  are  transient  and  temporary.  Transient  and  temporary 
faults  are  the  result  of  unexpected  conditions  in  input  data, 
unusual  or  unexpected  environment,  timing  incompatibilities  or 
equipment  incompatibilities.  These  occurrences  result  in  temporary 
equipment  problems  and  may  be  the  forecast  of  permanent  failures 
that  will  happen  after  more  equipment  operation. 

The  current  methods  of  BIT  or  testability  analysis  are  for 
the  most  part  based  upon  the  output  of  the  reliability  failure 
inodes  and  effects  analysis.  In  many  FMEAs  only  permanent 
failures  are  examined.  Some  make  an  attempt  to  examine  the 
temporary  faults,  but  in  most  cases  the  quantity  of  unexpected 
faults  is  significant  when  compared  to  the  fault  categories 
that  are  addressed  by  the  analysis. 

i.  Testability  Verification  and  Testing.  After  the  equipment 
is  produced,  there  is  a  need  for  verification  of  the  BIT  design. 
Even  prior  to  the  beginning  of  equipment  production,  there  is  an 
opportunity  to  conduct  equipment  simulations  to  evaluate  the 
status  of  the  BIT  design.  Verification  should  continue  through 
monitoring  of  the  actual  performance  of  the  BIT  as  the  system  is 
used  and  tested.  Because  the  normal  system  operation  may  not 
provide  the  quantity  of  faults  and  failures  that  are  required  to 
validate  the  BIT  there  should  also  be  an  assessment  based  upon 
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the  insertion  of  faults.  The  magnitude  of  this  program  and  the 
method  used  to  generate  the  list  of  faults  to  be  inserted  determines 
the  worth  of  this  program.  To  insert  failure  modes  that  are 
expected  and  well  understood  will  only  give  a  partial  assessment 
of  the  performance  of  the  BIT  in  operation.  The  fault  conditions 
that  are  experienced  in  the  "real  world"  may  be  quite  different 
from  those  generated  in  engineering  analysis.  This  is  one  reason 
why  most  BIT  development  programs  require  a  period  of  BIT  maturation 
during  early  operational  use. 

4.  Production 


In  the  production  area,  two  elements  were  examined.  These 
were  environmental  stress  screening  of  parts  and  equipment  and 
the  use  of  failure  reporting,  analysis,  and  corrective  action 
systems. 

a .  Environmental  Stress  Screening  of  Parts  and  Equipment. 

An  environmental  stress  screening  (ESS)  program  can  substantially 
reduce  the  number  of  latent  defects  in  parts  and  defects  due  to 
workmanship  in  delivered  hardware.  ESS  programs  were  effectively 
applied  on  the  aircraft  radar  programs  to  address  defects  at  the 
parts,  assembly,  box  and  system  level. 

The  use  of  ESS  as  part  of  the  receiving  inspection  of  piece 
parts  can  be  justified  economically  for  some  part  types.  Engin¬ 
eering  judgment  and/or  evaluation  must  be  made  as  to  which  specific 
screens  will  be  most  effective  and/or  necessary  for  specific 
part  types.  The  receiving  screens  supplement  screening  programs 
that  are  normally  applied  at  the  part  manufacturers. 

With  the  limited  historical  experience,  it  is  important 
for  screening  programs  to  remain  flexible.  Control  of  screening 
details  should  allow  for  adjustments  as  production  proceeds, 
with  government  control  of  some  final  result  (system  level 
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burn-in  with  a  failure-free  period  might  be  such  a  criterion). 

It  may  be  necessary  for  screens  used  during  the  assembly  of 
development  units  to  be  structured  for  a  different  class  of 
defects  than  those  for  high-rate  production. 

b .  Failure  Reporting/  Analysis  and  Corrective  Action  System. 
The  use  of  failure  reporting,  analysis  and  corrective  action 
system  (FRACAS)  is  an  essential  part  of  development  and  of 
reliability  growth.  Problems  or  failures  must  be  documented, 
analyzed  and  corrective  action  taken  for  reliability  growth  and 
maintainability  improvement  to  occur. 

Problems  or  failures  manifest  themselves  in  a  variety  of 
ways.  These  include  catastrophic  failures  and  out-of-tolerance 
drift.  Causes  can  include  operator  error,  overstress,  handling 
damage,  spontaneous  failure  of  piece  parts,  aging,  and  degradation 
Appropriate  effort  is  required  for  recognition,  diagnosis  and 
correction  of  these  failures  at  the  earliest  possible  phase  in  a 
product's  life-cycle.  Normally,  FRACAS  implementation  should 
start  about  mid-demonstration  and  validation  phase  and  extend 
through  FSD  and  production.  Cost  in  dollars  and  delays  increases 
by  orders  of  magnitude  if  the  defect  is  allowed  to  manifest 
itself  at  higher  levels  of  assembly  or  later  in  the  life-cycle. 

Potential  problem  identification  and  resolution  is  the  most 
productive  effort  because  the  design  can  be  changed  more  easily 
before  it  is  released  for  production.  Failures  must  be  documented 
in  a  closed-loop  tracking  system  to  assure  that  all  events  are 
given  appropriate  attention  and  corrective  action  is  considered. 

5.  Test  and  Evaluation 


In  the  test  and  evaluation  area,  five  elements  were  examined. 
These  five  elements  were  design-limited  qualification  testing, 
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reliability  growth  testing,  demonstration  testing,  operational 
test  and  evaluation,  and  in-service  assessment.  The  FRACAS  system 
must  be  used  for  all  tests  to  provide  adequate  feedback  to  the 
design  engineer. 

Most  testing  can  be  structured  to  provide  valuable  R&M 
information.  To  the  maximum  extent  practical,  every  test  should 
be  viewed  as  an  opportunity  to  assess  and  improve  R&M.  Certainly, 
from  the  time  that  any  reasonably  representative  hardware  is  built 
and  tested,  reliability  and  built-in-test  data  ought  to  be  collected, 
analyzed  and  acted  upon.  This  includes  development  tests  of  all 
types  as  well  as  various  qualification  tests.  Differences  in  equip¬ 
ment  configurations  and  test  environmental  conditions  will  compli¬ 
cate  assessments,  but  these  problems  do  not  outweigh  the  value  of 
the  opportunities  to  assess  and  mature  R&M. 

a.  Total  Program  for  Reliability  Growth.  A  well-structured 
reliability  growth  program  can  be  very  effective  in  the  development 
and  production  of  a  reliable  system.  Good  design  practices 
cannot  totally  eliminate  problems.  A  growth  program  to  identify 
problem  areas  and  develop  changes  to  grow  the  equipment  reliability 
and  substantiate  improvements  is  necessary.  Different  tests  in 
the  development  program  produce  different  data  as  far  as  failure 
modes,  environmental  conditions,  operating  modes  and  configuration. 

A  reliability  growth  program  should  use  data  from  all  test  and 
operational  sources.  The  program  should  be  structured  at  the 
beginning  to  treat  every  failure  or  operating  anomaly  as  a  poten¬ 
tial  reliability  growth  opportunity.  A  single  program  should 
examine  data  from  engineering  development  tests,  integration 
tests,  qualification  tests,  reliability  demonstrations,  produc¬ 
tion  tests  and  operational  tests.  The  growth  program  should 
continue  into  early  field  operation  with  contractor  personnel 
at  the  operational  sites.  Resources  and  organizational  struc¬ 
ture  must  exist  to  pursue  the  results  of  the  growth  program 
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data  analysis.  Additional  discussion  on  a  total  growth  program 
is  contained  in  the  main  report. 


an 


b.  Design  Limit  Qualification  Tests.  Design  limit  qualifi¬ 
cation  tests  can  be  used  to  obtain  R&M  data.  Qualification 
tests  expose  the  design  to  conditions  and  operating  modes  that 
may  not  have  been  seen  previously.  Engineering  data  from  these 
tests  provide  information  on  operations  in  these  environments. 

This  test  phase  is  usually  not  thought  of  as  an  R&M  element. 
Seme  programs  used  qualification  test  data  for  reliability 
and  maintainability  engineering  purposes.  These  tests  examine 
the  capability  of  the  design  (normally  one  sample)  to  meet  the 
performance  requirements  at  the  design  limit  (normally  for  a 
short  period  of  time  or  for  a  short  performance  test).  The  inter¬ 
relationship  and  sequencing  of  these  tests  and  R&M  tests  can  be 
critical  to  assure  that  the  final  design  is  adequately  tested. 

The  failure  data  (and  diagnostic  data)  from  these  tests  can  be 
valuable  since  it  will  be  some  of  the  earliest  data  available. 
Design  changes  made  later  in  the  program  must  be  evaluated  as  to 
their  impact  on  the  results  of  the  design  limit  tests. 

c.  R&M  Development  Testing.  A  well-structured  growth  or 
maturation  program  can  be  very  effective  in  the  development  and 
production  of  a  reliable  and  maintainable  system.  A  growth 
program  to  identify  problem  areas,  develop  changes,  and  substan¬ 
tiate  improvements  is  necessary.  Control  of  equipment  configura¬ 
tion  and  test  environment  may  require  dedicated  growth  test 
articles.  This  will  probably  be  true  of  avionics  equipment. 
However,  for  equipment  such  as  fixed  ground  electronics  it  may 

be  possible  to  combine  the  necessary  features  of  a  growth  program 
with  other  testing. 


826/1-53 


B-54 


•X 

k 

3 

\7 

*/> 


•v/r 


m 


3 


V.‘ 

p 


(1) .  Timing  of  Reliability  Development  Tests.  Relia¬ 
bility  development  testing  is  more  valuable  if  conducted  relatively 
early  in  a  development  cycle.  The  relative  contribution  of  relia¬ 
bility  growth  testing  is  a  direct  result  of  where  the  growth  test 
is  placed  in  the  sequence  and  timing  of  the  tests. 

Reliability  development  testing  can  be  a  very  effective 
task  in  the  development  of  reliable  complex  equipment.  This 
testing  should  be  viewed  as  a  "designer  tool"  to  provide  informa¬ 
tion  on  the  design's  capabilities,  identification  of  problem  areas 
and  effectiveness  of  corrective  actions  taken.  If  performed  early 
in  the  development  cycle,  this  test  can  provide  relatively  large 
amounts  of  reliability  information  in  a  defined  environment  with 
good  engineering  controls. 

If  the  reliability  development  test  is  late  in  the  schedule, 
then  the  data  may  be  of  limited  value  because  of  the  magnitude 
of  data  obtained  from  other  sources. 

The  test  plan,  environments,  number  of  test  units,  equipment 
operating  sequence,  procedures  for  failure  analysis  and  corrective 
action,  time  phasing  with  the  rest  of  the  development  program  are 
all  critical  to  the  contribution  of  the  growth  test.  Management's 
interest  in  the  progress  of  the  growth  test  can  be  a  positive 
factor  in  keeping  the  growth  test  on  schedule. 

(2) .  BIT  Development  Tests.  The  maturation  of  the  BIT 
system  is  dependent  upon  acquiring  engineering  data  on  the  actual 
faults  that  the  system  is  experiencing,  the  opportunity  to  elimi¬ 
nate  these  faults  and  the  potential  for  the  BIT  system  to  detect 
and  isolate  these  particular  faults  if  required,  and  the  elimi¬ 
nation  of  "unnecessary  indications"  of  failures  or  faults  that 

do  not  affect  the  system  performance.  The  length  of  time  required 
for  this  maturation  program  depends  on  the  quality  of  the  data 
being  collected.  When  compared  to  reliability  growth  testing, 
which  used  test  hours  as  its  basis,  BIT  development  testing 
should  use  the  number  of  faults  experienced  as  its  basis. 
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The  quality  of  the  data  also  depends  upon  the  realism  of  the 
operation  and  environment  that  the  system  is  exposed  to  during 
the  maturation  phase.  If  significant  areas  of  the  operating 
envelope  are  not  examined ,  then  many  of  the  potential  faults 
will  not  be  identified  or  corrected.  Every  opportunity  to  exercise 
and  evaluate  BIT  should  be  used  in  this  process.  Nevertheless, 
program  results  to  date  indicate  that  strong  consideration  should 
be  given  to  extending  planned  BIT  maturation  into  the  operational 
phase.  Presently,  the  data  collection  system  for  operational 
service  is  not  adequate  to  support  BIT  maturation,  making  correc¬ 
tion  of  BIT  problems  very  difficult.  A  planned  operational 
effort  including  special  data  collection  could  be  very  valuable. 

d.  Demonstration  Testing.  The  purpose  of  demonstration 
testing  is  to  provide  a  statistically  significant  measure  of 
compliance  with  specified  R&M  requirements.  Demonstrations 
should  be  included  as  an  integral  part  of  a  comprehensive  test 
program  designed  to  provide  leverage,  not  only  for  design  improve¬ 
ments  through  incorporation  and  verification  of  corrective  actions, 
but  also  for  the  basic  design  effort  by  the  contractor  knowing 
that  the  demonstration  will  be  run  and  must  be  passed.  Demonstra¬ 
tion  requirements  increase  management  visibility  and  emphasis  on 
attainment  of  specified  values.  Reliability  demonstration  during 
development  should  be  followed  by  demonstrations  of  production 
reliability  to  assure  correction  of  production  problems  not 
related  to  development,  and  attainment  of  higher  specified 
reliability  requirements  consistent  with  production  hardware. 

The  procedures  selected  for  demonstrations  should  be  con¬ 
sistent  with  the  R&M  program  risks.  For  example,  a  ground  electron¬ 
ics  program  using  mature  technology,  in  a  well-defined  environment, 
with  a  growth  program  and  a  reliability  improvement  warranty  may 
warrant  only  a  period  of  carefully  controlled,  all  equipment  data 
collection  and  subsequent  analysis.  On  the  other  hand,  customer 
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risks  associated  with  a  complex,  airborne  fire  control  radar 
would  probably  warrant  the  use  of  dedicated  assets  under  more 
controlled  conditions  for  the  data  collection  effort. 

In  any  event,  to  obtain  the  maximum  R&M  benefit,  demonstra¬ 
tions  must  be  tied  to  other  development  activities  such  as  a  closed- 
loop  failure  reporting  and  correction  system  to  provide  feedback 
to  the  designer  so  that  changes  can  be  implemented  to  improve  the 
design.  Demonstration  tests  should  be  integrated  into  a  total 
growth  program.  Additionally,  the  selection  of  procedures  to  be 
used  for  production  demonstrations  should  include  consideration  of 
the  timeliness  of  the  measurement,  e.g.,  a  lot  of  equipment  with 
problems  may  be  delivered  to  the  customer  if  tests  are  lengthy  and 
infrequent . 

e.  Operational  Testing.  Operational  testing  is  the  means  by 
which  weapon  system  performance  and  supportabili ty  characteristics 
are  evaluated.  Weapon  system  operational  effectiveness  and  opera¬ 
tional  suitability  including  R&M  factors  should  be  evaluated 
numerically.  The  detailed  results  from  operational  testing  must  be 
fed  back  to  the  design  team. 

f.  In-Service  R&M  Assessment.  In-service  assessment  and 
corrective  actions  have  been  shown  to  be  essential  in  achieving 
the  R&M  potential  of  the  design.  From  the  case  studies  examined, 
most  contractors  initiated  a  field  monitoring  program  to  assist  in 
identifying  problems  and  their  cause. 

Once  a  system  is  introduced  into  the  operational  environment, 
the  environment  (including  operating  and  maintenance  personnel) 
and  full-rate  production  may  introduce  a  new  set  of  R&M  problems 
that  must  be  identified,  analyzed,  and  corrected.  These  early 
production  systems  usually  result  in  a  large  number  of  engineering 
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changes.  In-service  assessment  required: 

-  Planning 

-  Assignment  of  engineering  resources  (including 
on-site  contractor  personnel) 

-  Selection  of  a  capable  R&M  data  system. 

Programs  that  did  not  plan  for  in-service  assessment  generally 
ended  up  initiating  such  a  program  because  of  demands  to  identify 
and  solve  field  problems.  Government  managers  must  recognize 
the  need  for  this  activity  and  provide  the  necessary  resources. 

In  summary ,  a  high  quality  of  R&M  data  should  be  acquired  in 
the  early  pre-deployment  and  deployment  time  periods  until 
acceptable  R&M  is  obtained.  This  can  be  achieved  in  a  reasonable 
period  of  time  by  an  aggressive  failure  analysis  and  corrective 
action  program.  For  already  fielded  systems ,  a  periodic,  compre¬ 
hensive  sample  data  collection  program  may  be  desirable. 
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C.  INTERRELATIONSHIPS  AND  DEPENDENCIES  OF  THE  ELEMENTS  FOR 

SUCCESSFUL  PROGRAMS 

The  purpose  of  this  section  is  to  discuss  how  R&M  elements 
interact  and  fit  together  to  support  operational  reliability  and 
maintainability.  Clearly,  these  case  studies  pointed  out  that 
combinations  of  elements,  not  individual  elements,  were  responsible 
for  the  program's  success.  An  example  in  point  is  that  a  good 
stress  analysis  individually  will  not  make  a  successful  program, 
but  its  omission  may  significantly  reduce  the  probability  of 
the  program  being  a  success  due  to  its  impact  on  other  program 
elements  reducing  their  effectiveness. 

To  ensure  that  the  overall  program  is  efficient  and 
effective  these  interrelationships  and  dependencies  must  be 
considered.  The  following  sections  provide  discussions  of 
interrelationships  and  dependencies  in  three  categories:  over¬ 
all  program  structure,  design,  and  schedule. 

1 .  Overall  Program  Structure 

A  typical  overall  program  structure  from  these  case  studies 
is  shown  on  Fig.  B-19.  A  program  starts  with  the  mission  and 
life  profiles  and  source  selection  criteria. 

Progression  down  the  chart  represents  a  time  sequencing  of 
events  for  an  overall  program.  Only  major  events  are  shown. 

Actual  elapsed  time  between  events  varies  from  program  to 
program,  but  the  relative  sequence  remains  similar.  Arrowheads 
represent  the  flow  of  information  to  complete  the  process. 

This  flow  is  not  a  new  revelation,  but  does  indicate  that  the 
same  basic  process  is  used  throughout  industry.  These  case 
studies  do  indicate  that  there  are  key  areas  of  interrelation¬ 
ships  and  dependencies  that  should  be  addressed. 
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First,  in  order  for  the  contractor  to  satisfy  the  customer, 
the  requirements  must  be  valid,  realistic  and  clearly  understood 
by  both.  If  this  is  not  true,  the  program  runs  a  high  risk  of 
not  satisfying  the  customer  needs.  In  the  case  where  the 
contractor  understands  what  he  is  told,  but  the  customer  does  not 
really  know  what  he  wants,  the  contractor  can  do  an  outstanding 
job  and  still  not  produce  an  acceptable  system.  In  the  other 
case,  where  the  contractor  does  not  understand,  he  has  little 
chance  of  producing  a  product  that  meets  the  customer's  require¬ 
ments. 

Establishing  realistic  mission  and  life  profiles,  LCC  and 
DTC  considerations,  quantitative  and  qualitative  requirements, 
and  source  selection  criteria  are  vital  to  promoting  understand¬ 
ing  and  acceptance  of  the  operational  R&M  requirements  by  both 
customer  and  contractor.  The  successful  programs  studied  bene¬ 
fited  from  this  process.  Of  special  note  are  the  T700  engine 
for  the  Blackhawk  helicopter,  which  led  to  a  highly  reliable 
and  maintainable  system;  the  F-16  radar,  which  has  surpassed 
operational  R&M  projections;  and  the  F-18  radar  which  has 
exhibited  high  initial  reliability. 

Second,  participative  management  and  individual  and  company 
motivation  play  a  major  role.  It  is  not  enough  just  to  accomplish 
certain  tasks.  The  right  tasks  must  be  done  well  and  completed 
at  the  right  time.  Planning,  control  and  management  emphasis, 
design  participation,  monitor/control  of  subcontractors  and 
suppliers,  process  control,  incentives,  and  CAD/CAM  all  contribute 
to  getting  the  right  things  done  well  and  on  time.  The  complexity 
of  the  acquisition  process,  combined  with  the  complexity  of  the 
systems  being  developed  present  a  formidable  challenge.  Incen¬ 
tives  can  play  a  major  role  in  achieving  high  operational  R&M 
provided  that  they  are  compatible  with  performance  requirements. 
They  will  provide  an  environment  to  motivate  the  contractor  to 
do  his  best. 
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Third,  program  risks  must  be  managed, 
a  continuous  assessment  process  must  be  initiated  and  problems 
identified  and  fixed  at  the  earliest  practical  time.  R&M 
failure  definitions,  dedicated  test  assets,  design  evaluation 
analysis/reviews,  design  limit  qualification  testing,  demonstration 
testing,  operational  test,  and  in-service  evaluation  are  all 
necessary  elements  of  this  assessment  process.  To  accomplish 
this  assessment,  a  comprehensive  growth  program  must  be  planned 
and  executed.  A  key  to  providing  R&M  growth  is  a  good  failure/ 
corrective  action  program.  All  analysis  and  evaluation  tools 
must  be  used  to  manage  and  control  program  risks. 

The  key  to  implementing  these  tools  is  a  R&M  growth  program 
that  includes  the  results  of  all  testing.  A  test-analyze  and  fix 
( TAAF )  philosophy  applied  throughout  the  program  and  continued 
into  field  operations  will  provide  the  maximum  benefit. 

Failure  definitions  which  form  a  part  of  the  standards 
against  which  progress  is  measured  must  be  carefully  constructed. 
Dedicated  assets  facilitate  design  verification  and  the  timely 
identification  and  correction  of  problems.  Design  evaluation 
analysis/review  promote  a  disciplined  design  process  and  provide 
for  early  incorporation  of  corrective  action  when  it  is  less 
expensive  to  make  changes.  Design  limit  qualification  testing 
provides  an  assessment  of  the  design  in  environmental  extremes. 

A  failure  analysis/corrective  action  program  will  provide  assess¬ 
ments  of  the  cause  of  problems  and  help  to  determine  what 
corrective  actions  should  be  incorporated.  This  also  aids  in 
establishing  the  effectiveness  of  the  corrective  actions.  The 
remaining  elements  assess  operational  R&M  in  progressively  more 
realistic  environments  and  usage.  Properly  used,  each  of  these 
elements  contributes  to  keeping  program  risks  low.  The  success¬ 
ful  cases  studied  used  these  elements  well.  They  identified 
most  problems  early  and  fixed  them  before  the  fixes  became  too 
expensive  to  implement. 
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These  cases  studied  showed  that  the  combination  of  areas 
discussed  above  was  what  made  the  programs  successful.  Neglect 
of  any  of  these  areas  invites  problems.  Decisionmakers  must 
recognize  the  interrelationships  and  dependencies  and  avoid 
arbitrarily  constructing  and  implementing  R&M  programs.  Selected 
elements  must  be  coordinated  to  ensure  that  both  the  customer 
and  contractor  clearly  understand  what  is  needed,  that  the 
process  is  well-managed  by  both  customer  and  contractor,  and 
that  the  assessment  tools  are  sufficient  to  provide  management 
the  information  necessary  to  control  program  risks. 

2.  Technical/Desiqn  Dependencies 

As  at  the  higher  level  of  consideration,  there  are  numerous 
important  interrelationships  between  the  R&M  related  activities 
in  the  design  process.  It  is  important  for  decisionmakers  to 
have  an  appreciation  for  the  extent  of  these  dependencies  when 
they  are  making  tough  decisions  dealing  with  the  allocation  of 
scarce  program  resources. 

Findings  from  these  case  studies  show  that,  in  general,  all 
the  analyses  identified  on  Fig.  B-20  were  accomplished  at  various 
phases  during  the  design  process.  In  some  cases,  the  analyses 
were  not  done  as  early  as  one  would  desire,  but  the  results 
were  available  to  influence  the  design  process.  Actual  detail 
timing  was  difficult  to  identify  since  many  of  the  analyses 
are  an  iterative  process.  It  was  concluded  that  most  of  the 
analyses  had  been  done  in  a  timely  manner  except  for  the  BIT 
and  ATE  analysis.  This  had  not  been  given  as  high  a  priority 
and  therefore  its  development  and  maturing  was  coming  later 
than  desired. 
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FIGURE  B-20.  Interrelation  on  Design  Elements 
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In  many  cases  decisionmakers  are  faced  with  resource 
shortages  and  are  tempted  to  cut  out  or  severely  restrict  some 
elements  of  the  R&M-related  design  activities.  Done  without  an 
understanding  of  the  interrelationships,  this  could  be  disastrous. 
Consider  the  following  illustration.  Reliability  predictions 
are  a  measure  of  a  design's  potential  and  are  used  to  assess 
capability  to  meet  allocated  design  requirements.  The  reliabil¬ 
ity  prediction  (Fig.  B-21)  requires  knowledge  of  part  selections 
and  stress  levels.  Hence,  a  good  prediction  requires  an  informa¬ 
tion  flow  as  depicted  by  the  arrowheads.  It  includes  a  stress 
analysis  which  is  in  turn  dependent  on  a  thermal  analysis  for 
component  temperature  information.  The  thermal  analysis  requires 
cooling  system  information  which  in  turn  must  have  environmental 
input  information  from  the  mission  and  life  profiles;  without 
the  stress  analysis,  the  reliability  prediction  would  require 
many  assumptions  which  could  seriously  impact  its  accuracy. 

Without  the  thermal  analysis,  the  stress  analysis  could  be 
significantly  in  error;  this  again  impacts  the  prediction  and 
the  design  itself. 

The  point  is  that  deletion  or  constraint  of  any  of  the 
related  elements  could  significantly  impact  the  outcome  of  the 
whole  effort.  Decisionmakers  must  explore  the  interrelation¬ 
ships  present  in  their  program  and  assess  the  probable  impacts 
and  consequences  prior  to  making  R&M  program  decisions. 


Schedule  Dependencies 


Equally  important  to  the  inclusion  of  mutually  supporting  R&M 
program  elements  is  the  scheduling  of  all  R&M  elements.  Timeliness 
of  information  may  be  critical  to  program  success. 
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In  general,  the  findings  from  the  case  studies  have 
indicated  an  acceptable  scheduling  of  efforts.  Figure  B-19 
shows  an  overall  sequence  for  the  major  activities. 

An  area  where  schedule  dependencies  is  very  important  is 
during  the  test  and  evaluation  phase.  In  the  case  of  the  F-18, 
the  reliability  development  tests  were  run  concurrently  with 
the  total  aircraft  full-scale  develooment  tests.  The  same 
generation  of  hardware  was  tested  in  both  programs.  Some 
additional  problems  were  identified  in  the  development  test, 
but  had  it  been  run  earlier  in  the  program,  the  full-scale 
testing  could  then  have  been  done  on  the  next  generation  hard¬ 
ware,  thus  giving  feedback  on  the  corrective  actions  found  and 
implemented  during  development  testing  and  identifying  problems 
on  the  next  generation  of  hardware  prior  to  delivery  to  the  fleet. 
Corrective  actions  found  during  the  concurrent  testing  were  not 
incorporated  until  later  deliveries.  Indications  from  these 
later  deliveries  show  the  APG-65  to  have  a  field  reliability 
approaching  40  hours  versus  the  24  hours  reflected  in  the  case 
studies.  Had  the  development  test  been  run  earlier,  this 
improvement  would  have  shown  up  during  the  full-scale  develop¬ 
ment  test  and  on  the  first  units  delivered,  but  the  fact  of 
test  concurrency  allowed  the  systems  to  be  fielded  earlier  than 
with  sequential  testing. 

Also,  in  considering  the  preceding  illustration  (Fig.  B-21), 
if  the  mission  and  life  profiles  are  not  defined  adequately 
prior  to  the  need  for  thermal  analysis,  the  thermal  analysis 
must  either  be  delayed  or  begun  using  assumptions  about  available 
cooling  capability.  If  the  thermal  analysis  is  delayed,  the 
other  efforts,  including  design  iterations,  must  either  be  delayed 
or  begun  using  assumptions. 

If  the  dependent  work  is  begun  or  completed  prior  to  the 
independent  work  which  feeds  the  system,  the  amount  of  dif¬ 
ficulty  later  encountered  is  a  function  of  the  accuracy  of  the 
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assumptions  made.  If  some  of  the  assumptions  are  significantly 
in  error,  management  is  faced  with  a  redesign  effort,  accepting 
the  consequences  or  something  in-between.  Many  times  the  cost 
and  schedule  consequences  of  redesign  would  probably  be 
unacceptable.  Therefore,  something  is  done  which  produces  less 
than  the  desired  results.  Additionally,  the  work  done  out  of 
sequence  uses  the  available  resources  ineffectively.  An  example 
of  this  is  the  failure  modes,  effects,  critically  analysis 
(FMECA).  When  this  is  done  off-line  after  PDR ,  it  is  unlikely 
to  significantly  affect  design  for  diagnostics  or  the  elimination 
of  single-point  failures. 

Ideally,  a  program  should  be  structured  such  that  needed 
information  is  timely  and  available.  In  practice,  despite 
proper  initial  scheduling,  information  flow  will  sometimes 
break  down  and  decisionmakers  will  have  to  decide  the  course  of 


action  to  be  taken  based  on  limited  facts.  There  is  no  set 


formula.  Each  decision  should  be  made  on  its  own  merit. 


However,  the  decisionmaker  should  always  strive  to  understand  the 
risks;  the  decisionmaker  should  explore  the  schedule  dependencies 
prior  to  making  a  decision. 
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D.  CONCURRENCY  AND  SCHEDULING 

I .  Introduction 


This  discussion  on  concurrency  is  limited  only  to  those 
aspects  relative  to  R&M.  The  degree  of  concurrency  in  a 
development  program  can  introduce  both  positive  and  negative 
factors.  These  factors  must  be  recognized  by  program  managers 
and  appropriate  management  action  taken  to  structure  a  program 
t^at  balances  any  increased  risk  with  compensating  program 
activities  or  resources. 

Concurrency  can  be  a  positive  factor  in  design  maturation 
by  resulting  in  earlier,  more  abundant  failure  data  under  the 
operational  environment.  One  negative  factor  can  be  the  reduced 
time  that  is  available  to  take  any  design  action  after  a  design 
analysis  or  test  before  the  design  is  frozen  for  production. 

In  first  generation  technologies  such  as  electrooptic  systems 
it  appears  that  the  impacts  may  be  more  difficult  to  balance  or 
may  compound  the  risks  associated  with  these  technologies.  In 
general  the  radars  examined  by  the  case  studies  represented 
second  or  third  generation  technologies  with  moderate  concurrency 
and  were  able  to  compensate  for  the  risks  and  show  improvements 
over  the  previous  generations.  The  risks  being  introduced  by 
concurrency  must  be  understood  and  balanced.  To  achieve  this 
balance,  detailed  planning  and  risk  assessment  is  a  mandatory 
function.  Readily  available  and  well-planned  options  must  be 
prepared  for  execution  in  the  event  concurrent  activities  clash 
and  create  program  disruptions.  As  discussed  in  Section  2, 
Appendix  P,  the  impact  of  a  funding  profile  to  provide  a  more 
advanced  system  prior  to  the  start  of  testing  will  reduce  some  of 
the  inherent  shortcomings  of  test  concurrency. 
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2.  Definition 


The  defining  of  concurrency  is  not  a  simple  matter.  The 
Defense  Science  Board  performed  a  study  on  concurrency  in  1977. 
From  that  study,  the  following  definition  was  developed: 

"The  conduct  of  the  steps  leading  to  production  for 
inventory  before  the  end  of  full-scale  development 
time  span." 

The  most  frequent  definitions  found  in  literature  on  the 
subject  included:  (a)  parallel  (back-up)  technological  develop¬ 
ment,  (b)  concurrent,  but  independent  subsystem  development  and 
testing,  (c)  co-production,  and  (d)  overlap  of  dependent,  normally 
sequential  activities. 


Concerns  of  Concurrency 


Use  of  the  concurrency  concept  on  a  program  has  both  advantages 
and  disadvantages  which  must  be  weighed  against  all  other  program 
functions.  If  the  risks  warrant  use  of  concurrent  actions,  then 
it  is  reasonable  to  limit  the  advantages/disadvantages  as  follows: 

•  Potential  advantages  include:  (a)  concurrency 
potentially  allows  the  attainment  of  an  earlier 

IOC,  (b)  increased  likelihood  of  meeting  intermediate 
goals  and  thresholds,  (c)  lower  overhead  costs, 

(d)  work  force  continuity,  and  (e)  increased  worker 
motivation. 

•  Potential  disadvantages  include:  (a)  concurrency 
may  lead  to  premature  commitment  to  high-cost 
program  elements,  (b)  excessive  and  high-cost 
changes  in  design  after  production  has  commenced, 

(c)  less  reliable  equipment  in  service,  and  (d)  degra¬ 
dation  of  training  because  of  multiple  configurations 
and  faulty  systems. 
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The  effect  of  concurrency  on  R&M  factors  in  a  program  can 
be  very  positive,  assuming  the  available  data  are  applied  correctly. 
The  benefits  gained  result  from  the  insight  to  problems  through 
feedback  from  one  action  to  another,  i.e.,  prototype  testing  to 
full-scale  development  during  design.  The  feedback  concept  is 
illustrated  in  Fig.  B-22.  The  provision  for  continuing  feedback 
to  design,  production  procedures,  parts  screening  or  evaluation, 
etc. ,  allows  R&M  factors  to  be  reviewed  and  evaluated  in  a  very 
near  real-time  fashion.  As  a  result,  changes  can  be  determined 
and  effected  sooner  to  make  improvements  where  required  and 
avoid  problems  at  a  later  time  in  the  program's  development. 

The  case  studies  performed  on  the  F-15/16/18  radar 
systems  demonstrated  how  concurrent  activities  played  a  role 
in  producing  systems  with  significantly  better  R&M  factors 
than  previous  generations. 

The  overlap  of  various  program  phases  all  appeared  to  contri¬ 
bute  to  the  development  of  a  better  level  of  R&M  (see  Figs.  B-23, 
B-24,  B-25  and  B-26).  Obviously,  planning  and  management  aware¬ 
ness  were  very  important  factors  in  achieving  these  gains.  It 
is  also  interesting  to  note  that  the  concurrent  activities  did 
not  coexist  with  production  for  a  prolonged  period  of  time. 
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4.  Scheduling  Factors 


The  sequencing  and  time  allocated  for  accomplishing  the 
tasks  associated  with  the  key  R&M  elements  are  critical  if  the 
R&M  requirements  are  to  be  met  when  production  hardware  is 
delivered  to  the  user.  Many  of  the  R&M  elements  are  interrelated 
and  should  be  properly  scheduled  to  design,  develop  and  build 
reliable  and  maintainable  systems. 

There  are  many  items  that  must  be  considered  when  developing 
a  concurrent  program  schedule,  some  of  which  may  result  in  less 
than  an  optimum  schedule  for  R&M  considerations.  The  key  items 
that  influence  concurrent  program  schedule  decisions  are  budget 
constraints,  urgency  for  system,  system  complexity  and  technology 
being  used.  Scheduling  considerations  for  the  various  R&M  elements 
will  be  discussed  in  subsequent  paragraphs  of  this  section.  The 
impact  that  budget/time  constraints  can  have  on  developing  success¬ 
ful,  from  an  R&M  viewpoint,  systems  is  also  discussed  along  with 
R&M  schedule  considerations  when  introducing  new/advanced  tech¬ 
nology.  It  is  assumed,  in  subsequent  paragraphs,  that  there  are 
three  phases  in  the  program(s) — concept  definition/validation, 
full-scale  development,  and  production. 

a.  Concept  Definition  and  Full-Scale  Development.  As 
indicated  above,  there  are  usually  three  phases  of  the  system 
acquisition  process.  R&M  element  considerations  start  at  the 
beginning  of  the  process  and  continue  into  production  as  shown 
in  Fig.  B-27. 

In  general,  the  R&M  requirements  and  design  criteria  elements 
should  be  established/ implemented  in  the  concept  definition  phase, 
and  the  R&M/design  evaluation  analysis  and  R&M  testing  in  the 
FSD  phase.  A  system  for  identification  and  investigation  of 
problems  should  be  used  throughout  the  acquisition  process.  The 
acquisition  period  through  FSD  will  vary  from  program  to  program. 
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The  data  presented  in  the  case  studies  indicated  a  six-  to  seven- 
year  cycle  for  the  radars  that  were  reviewed  (see  Fig.  B-26)  and 
ten  years  for  the  T700  engine  (See  Fig.  B-28). 

Failure  to  establish  requirements  and  design  criteria 
prior  to  start  of  detail  design  will  stretch  out  the  design 
period  and/or  may  require  redesign  at  a  later  date.  It 
should  be  noted  that  even  though  work  is  started  on  these 
elements  prior  to  FSD,  it  is  necessary  to  continue  and  update 
the  data  throughout  the  design  period. 


(1).  Scheduling  of  R&M/Design  Evaluation  Analysis. 
Trade  studies,  update  of  R&M  models,  environmental  studies/ 
analysis,  derating  criteria  and  R&M  predictions  and  allocations 
are  continually  performed  throughout  the  detailed  design  period. 
The  R&M  design  evaluation  analysis  (thermal  analysis,  stress 
analysis,  FMEA,  worst-case  analysis  and  sneak-circuit  analysis) 
must  be  an  integral  part  of  the  detailed  design  process  and 
continue  to  be  updated  through  CDR  and  as  required,  when  changes 
are  implemented.  These  R&M  design  analyses  need  to  be  conducted 
concurrent  with  the  detailed  design  so  that  the  analysis  results 
can  cause  design  changes  in  a  timely  manner.  While  most  of 
these  analyses  are  normally  associated  with  electric/electronic 
design,  similar  type  analyses  are  applicable  to  and  done  for 
mechanical  systems. 

Scheduling  of  PDRs  and  CDRs  are  dependent  on  complexity  of 
system,  technology  being  used,  extent  of  front-end  design,  devel¬ 
opment  work  planned  and  the  urgency  for  the  system.  The  period 
of  time  between  start  of  FSD  to  CDR  varied  from  four  months  for 
the  F-16  radar  to  eleven  months  for  the  F-18  radar.  Engineering 
models  of  the  F-15  and  F-16  had  been  designed,  fabricated  and 
tested  during  the  concept  definition  (demonstration)  phase  of 
the  program  so  a  lot  of  the  design  tasks  had  been  accomplished 
prior  to  the  start  of  FSD.  For  design  evolution  systems  such 
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as  the  P-15,  F-16,  and  F/A-18  radars  about  one  year  from  FSD 
turn-on  to  CDR  is  adequate  to  accomplish  the  design  task. 


(2).  Scheduling  of  Tests  for  R&M.  Good  front-end 
engineering  for  R&M  is  necessary  for  providing  reliable  and 
maintainable  hardware  to  the  user  but  there  is  no  way  all  of  the 
R&M  problems  can  be  anticipated  and  designed  out.  As  the  program 
transitions  from  engineering  model  hardware  to  prototype  systems 
to  production  to  the  field,  new  environmental,  skill,  tooling, 
process  and  usage  conditions  are  encountered  that  will  identify/ 
introduce  new  problems.  At  each  of  these  points,  testing  and 
design  changes  are  often  required.  The  testing  that  is  accomplished 
during  the  demonstration/validation  phase  and  engineering  devel¬ 
opment  portion  of  the  FSD  is  primarily  for  technical  performance 
evaluation.  The  hardware  used  in  these  tests  is  usually  engi¬ 
neering  developmental  models  and  not  representive  of  the  final 
design.  The  hardware  fabricated  during  FSD  is  representative  of 
the  CDR  design  but  usually  is  not  fabricated  with  the  controlled 
processes  and  tooling  that  will  be  used  during  production.  The 
configuration  of  the  production  hardware  will  probably  not  be 
the  same  as  the  FSD  hardware  due  to  changes  required  as  a  result 


of  the  FSD  testing. 

Because  of  the  evolution  of  the  hardware  configuration,  the 
formal  reliability  and  maintainability  qualification  tests  or 
demonstrations  should  be  scheduled  late  enough  in  FSD  to  assure 
that  the  configuration  is  as  close  as  possible  to  the  production 
configuration  but  early  enough  to  provide  for  timely  corrective 
action.  Reliability  development  growth  testing  scheduled  prior 
to  demonstration  testing  can  be  used  to  identify  corrective  action 
or  early  FSD  configurations  and  allow  demonstration  testing  to 
be  accomplished  on  later  configurations  with  corrective  action 
incorporated. 
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Data  from  all  design,  development  and  production  tests 
should  be  used  to  determine  design  changes  that  will  improve 
the  reliability  and  maintainability  of  the  hardware.  Thus, 
the  importance  of  feedback  from  concurrent  actions  comes  into 
play. 


(a)  .  Failure  Reporting,  Analysis  and  Corrective  Action 
( FRACAS ) .  Reliability  growth  during  the  acquisition  cycle  is 
dependent  on  identifying  and  correcting  problems.  Identifying 
and  correcting  the  problems  as  early  in  the  cycle  is  desired  to 
assure  early  incorporation  of  fixes.  A  failure  reporting,  analysis 
and  corrective  action  system  should  be  implemented  during  the 
concept  definition  phase  and  used  for  the  complete  acquisition 
cycle  and  during  early  field  use.  Here  again,  some  degree  of 
schedule  overlap  is  important,  and  again  concurrency  becomes  an 
element  in  management  of  the  program. 

(b) .  Reliability  &  Maintainability  Growth  Program.  A 
well-planned  R&M  growth  program  is  very  effective  in  the  devel¬ 
opment  and  production  of  a  reliable  system.  The  program  should 
be  structured  to  use  data  from  development,  integration,  qualifi¬ 
cation,  reliability  qualification,  production  and  operational 
tests.  A  reliability  growth  test  with  dedicated  assets  should 

be  included  in  the  reliability  growth  program.  A  reliability 
growth  test  would  provide  a  lot  of  test  hours  in  a  controlled 
environment.  This  should  be  scheduled  early  in  the  FSD  program 
so  that  any  changes  could  be  incorporated  in  production  hardware. 
The  duration  of  the  growth  tests  should  be  determined  by  corrective 
actions  generated  instead  of  a  fixed  number  of  operating  hours. 

If  dedicated  assets  for  reliability  growth  testing  are  not 
available  due  to  cost  considerations,  the  growth  testing  could 
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be  combined  with  the  qualification  tests.  In  fact,  failure  data 
from  nearly  all  testing  can  be  growth  data  if  careful  analysis 
is  performed.  The  risk  of  relying  only  on  data  from  other  tests 
is  that  all  of  the  design/operational  envelope  conditions  may 
not  be  encountered  early  in  the  development  program  and  some 
problems  not  detected  until  a  later  time  period. 

(c) .  Thermal  Verification  Tests.  One  important  factor 
affecting  reliability  is  good  thermal  management.  In  all  of  the 
case  studies,  a  thermal  analysis  was  performed  during  the  design 
to  optimize  the  design  of  the  hardware  and  thermal  verification 
test  conducted  to  evaluate  the  design.  These  tests  should  be 
conducted  as  early  as  possible  so  that  required  changes  may  be 
implemented  prior  to  production.  Caution  should  be  exercised  to 
ensure  that  tests  are  not  run  until  the  system  operational  envi¬ 
ronment  has  been  verified. 

(d) .  R&M  Qualification  (Demonstration)  Test.  The 
problem  of  starting  reliability  and  maintainability  demonstration 
tests  too  early  on  unrepresentative  configurations,  versus  too 
late  on  production  equipment,  presents  a  dilemma  typical  of  all 
testing.  Early  testing  involves  additional  risks  of  failing 

the  required  R&M  test  requirements  versus  later  testing  when  the 
ability  to  incorporate  early  production  fixes  is  minimized. 

When  development  growth  tests  are  specified.  Reliability 
and  Maintainability  demonstration  tests  may  be  scheduled  later 
in  the  FSD  program  to  assure  that  the  configuration  is  as  close 
as  possible  to  that  of  production  hardware.  When  development 
growth  tests  are  not  specified,  demonstration  tests  must  be 
conducted  early  to  provide  for  timely  corrective  action.  Demon¬ 
stration  testing  scheduled  late  in  the  FSD  program  without 
prior  reliability  development  testing  is  of  less  value  to 
the  designer. 
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When  many  units  are  to  be  produced,  but  initial  production 
rate  is  low,  it  may  be  acceptable  to  use  the  first  production 
units  for  reliability  demonstration  if  there  has  been  an  aggressive 
reliability  growth  program. 

b.  Production 

When  going  from  Full-Scale  Development  into  production, 
the  hardware  faces  a  different  environment,  different  people, 
different  skill  levels  and  different  processes.  To  assure  that 
Reliability  has  not  been  degraded  with  this  transition,  a 
production  reliability  test  should  be  conducted  early  in  the 
production  cycle.  To  identify  any  degradation  in  reliability, 
production  reliability  tests  should  be  done  on  a  periodic  basis. 

The  failure  reporting,  analysis  and  corrective  action 
system  developed  during  the  development  phase  should  be  carried 
on  into  the  production  phase  and  used  for  production  acceptance 
testing  and  production  reliability  program.  The  data  obtained 
can  be  used  for  reliability  growth  during  the  production  cycle. 

BIT/BITE  should  be  used  to  the  maximum  extent  possible 
during  production  testing.  FRACAS  data  obtained  in  this  manner 
can  be  used  to  improve  system  maintainability.  The  F-16  program 
has  been  very  successful  in  achieving  reliability  growth  during 
production. 

By  employing  a  reasonable  level  of  concurrent  activities 
between  production  and  FSD,  the  feedback  will  allow  timely 
corrections  to  be  made  that  will  reduce  degradation  to  the 
R&N  factors  of  the  system. 

(1).  Impact  of  Concurrency.  As  shown  in  Fig.  B-26, 
the  F-15,  F-16,  and  F/A-18  programs  were  highly  concurrent  in 
that  production  go-ahead  was  before  FSD  completion.  On  the 
F-15,  production  go-ahead  was  one  month  after  CDR  and  14  months 
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before  the  first  prototype  system  was  completed.  On  the  F-16 
and  F/A-18  programs,  production  go-ahead  was  approximately  one 
year  after  CDR,  and  on  the  F-18  program  before  the  first  prototype 
system  was  completed.  The  first  production  radars  were  delivered 
approximately  1-1/2  years  before  FSD  completion.  The  AN/TPQ-37 
( FIREFINDER)  ground-based  radar  went  from  Advanced  Development 
phase  to  production. 

The  T-700  Turbine  Engine  program  was  the  classic  non¬ 
concurrent  type  program.  The  T-700  program  structure  included 
a  4-year  (approx.)  Advanced  Turbine  Engine  Demonstration  Program, 
followed  by  a  4-year  FSD  program.  Production  go-ahead  was  after 
completion  of  FSD.  The  FSD  phase  was  followed  by  a  post  quali¬ 
fication  maturity  program.  The  maturity  program  resulted  in  a 
smooth  transition  to  production  and  a  significant  reduction  in 
design  changes  during  the  early  years  in  the  field. 

The  data  from  the  above  case  studies  indicate  that  concurrency 
did  not  have  an  appreciable  impact  on  FSD  schedules.  The  data 
does  indicate,  as  one  would  expect,  that  all  the  R&M  problems  were 
not  identified  prior  to  delivery  of  production  hardware,  and 
it  was  necessary  to  incorporate  retrofit  changes  in  delivered 
hardware  to  meet  R&M  requirements. 

It  is  believed  that  R&M  requirements  can  be  achieved  at  an 
earlier  date  in  a  concurrent  program  than  in  a  non-concurrent 
program  if  the  program  is  structured  properly.  Good  front-end 
engineering  and  growth  programs  are  required  to  achieve  reliable 
and  maintainable  hardware.  The  funding  profile  proposed  earlier 
in  this  report  will  help  this.  However,  as  discussed  in  previous 
sections,  new  problems  are  encountered  when  the  program  transitions 
from  design  to  production  and  when  it  is  first  used  by  the  opera¬ 
tional  forces.  Since  these  events  occur  at  an  earlier  date  on  a 
concurrent  program,  the  problems  identified  after  transitions  are 
found  earlier.  To  realize  R&M  improvements  the  program  has  to  be 
structured  for  an  R&M  growth  program  extending  into  production  and 
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early  field  deployment.  Contractor  engineering  support  in  the 
field  would  be  required  for  operational  problems,  understanding, 
feedback,  and  correction.  To  realize  R&M  improvements  in  a  timely 
fashion  the  flow  time  for  approval  of  Class  I  changes  needs  to  be 
improved.  The  reliability  growth  experienced  on  the  F-16  and  TPQ-36 
and  TPQ-37  radar  during  production  and  operational  deployment 
is  an  example  of  such  a  reliability  growth  program. 


(2).  Observations  Related  to  Concurrency.  In  summary, 
for  concurrent  programs  several  factors  must  be  accounted  for 
and  specific  functions  completed  by  management. 

A  program  with  concurrency  applied  must  be  well-planned. 

The  funding  profile  for  the  total  program  must  include  earlier 
ccnunitment  of  funds  to  provide  a  more  mature  design  prior  to 
the  testing  phase.  Alternate  paths  must  be  identified  early  to 
assist  in  avoiding  delays  should  problems  arise  due  to  the  level 
of  concurrency  employed  or  for  any  other  reason.  Schedules 
must  be  laid  out  and  maintained  in  order  to  ensure  the  necessary 
milestones  are  met. 

Prior  to  implementation  of  concurrency  in  a  program,  all 
related  risks  must  be  reviewed  and  assessed.  R&M  benefits 
are  just  two  of  the  many  factors,  and  should  by  no  means  be  the 
only  driver. 

The  testing  data  and  how  they  will  be  fed  into  the  design, 
production  and  evaluation  activities  must  be  thoroughly  understood 
and  articulated  to  all  levels  of  management.  Checks  and  balances 
must  be  employed  to  be  sure  concurrency  is  providing  the  desired 
results. 
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Analyzing  Program  Schedule 


One  of  the  first  steps  in  the  acquisition  of  new  systems  is 
the  process  of  selecting  an  acquisition  strategy  and  establishing 
the  program  schedule.  The  strategy  selected  will  depend  on  a 
variety  of  factors  in  order  to  take  into  account  when  the  system 
is  needed  (i.e.,  the  threat),  the  capability  of  existing  systems, 
political  considerations,  cost,  technological  maturity,  etc. 

The  program  schedule  is  then  developed  considering  those  factors. 
Numerous  Service  and  independent  agency  studies  have  addressed 
this  process.  One  of  the  more  recent  of  these  studies,  the  Air 
Force's  "Affordable  Acquisition  Approach  (A-*)  Study"  focused 
heavily  on  this  front-end  part  of  the  acquisition  process  and 
highlighted  the  importance  of  realistic  program  baselines-- 
getting  off  on  the  right  foot.  The  question  is  how  can  we  do 
it? 

The  purpose  of  this  section  is  to  present  a  prototype  tool 
for  analyzing  program  schedules.  This  tool,  and  others  to  be 
developed,  used  in  conjunction  with  existing  tools  such  as  "should 
cost  analysis"  can  be  evolved  into  a  set  of  expert  judgment  aids 
to  improve  the  baselining  process. 

Tools  and  analytical  approaches  are  required  in  order  to:  (1) 
develop  acquisition  program  plans  which  establish  attainable 
program  baselines,  i.e.,  a  reasonable  balance  of  design  time, 
environmental  testing,  field  maturation,  funding  levels,  etc., 

(2)  provide  a  reasonable  capability  to  access  program  factors 
during  the  front-end  process  as  well  as  throughout  the  development 
process  as  program  contingencies  force  changes  to  the  baseline 
plan.  An  example  of  such  a  tool  and  an  analytic  approach  has 
been  developed  for  airborne  fire  control  radars  which  if  refined 
and  tested  can  be  used  as  a  pattern  for  other  types  of  systems. 

The  fire  control  radar  example  was  developed  using  information 
from  the  case  studies  for  the  APG-63  (F-15),  APG-65  (F/A-18), 
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and  the  APG-66  (P-16),  information  from  the  Air  Force  maintenance 
data  system  (DO-56)  and  a  study  for  the  Air  Force  by  General 
Electric.  The  GE  study,  "Research  Study  of  Radar  Reliability 
and  its  Impact  on  Life  Cycle  Costs,"  provides  information  for 
the  APO-113  (F-111A  and  F-111E),  APO-114  (FB-111A),  APO-144 
(F-111F)  and  the  APQ-120  (F-4E).  Table  B-l  presents  the  data  used 
to  discuss  the  example. 

The  concept,  using  the  fire  control  radar  as  an  example, 
establishes  a  standard  way  of  presenting  information  on  a  class 
of  like  systems  so  that  "red  flags"  could  be  raised  on  new  systems 
when  certain  parameters  are  identified.  That  is,  historical 
information  on  a  similar  class  of  radars  may  be  used  to  highlight 
areas  for  further  investigation.  Figure  B-29  represents  one 
such  approach.  In  this  example,  the  number  of  electrical  parts 
in  each  radar  is  plotted  versus  the  number  of  months  from  contract 
go-ahead  to  the  delivery  of  the  first  production  unit.  The 
number  of  parts  is  used  as  a  surrogate  for  complexity  because  it 
tends  to  be  a  measure  which  equates  the  magnitude  of  the  design 
and  engineering  task  between  systems  and  because  it  tends  to  do 
so  regardless  of  the  technology  used  in  the  system.  For  example, 
the  APG-65  and  the  APQ-120  contain  about  the  same  number  of 
parts  but  are,  from  a  technology  view  point,  very  different, 
e.g .,  172  integrated  circuits  (ICs)  in  the  APQ-120  versus  5,329 
ICs  in  the  APG-65.  The  APG-65,  developed  ten  years  after  the 
APQ-120,  is  also  a  far  more  capable  radar.  Even  so,  the  task 
of  integrating  13,500  parts  into  a  working  radar  can  be  assumed 
to  be  equivalent  for  our  purposes. 

The  second  variable  plotted  on  Fig.  B-29  is  the  number  of 
months  from  contract  go-ahead  to  delivery  of  the  first  production 
radar.  Intuitively,  it  should  require  incrementally  more  time 
to  develop  and  deliver  a  radar  as  complexity  (number  of  parts) 
increases.  The  next  step  was  to  draw  a  line  to  represent  the 
relationship  of  complexity  to  time,  i.e.,  as  the  number  of  parts 
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Months  After  Go-Ahead  to  First  Production  Delivery 


FIGURE  B-29.  Airborne  Fire  Control  Radar  Development — Number  of 
Months  from  Go-Ahead  to  First  Production  Delivery 
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increases,  how  much  additional  development  time  is  required. 

Before  proceeding,  it  is  important  to  understand  what  this 
analysis  can  and  cannot  be  expected  to  do.  The  purpose  of  the 
tool  and  the  supporting  analysis  process  is  not  to  judge  a 
program  schedule  as  good  or  bad  but  instead  to  judge  the  balance 
between  time  and  complexity  when  other  program  factors  and  develop¬ 
ment  techniques  are  reviewed  for  reasonableness.  Table  B-2  is  a 
matrix  for  comparison  of  the  factors  used  in  conjunction  with 
Fig.  B-29  to  develop  the  prototype  analysis  approach.  With  an 
understanding  of  what  the  tool  can  be  expected  to  do  and  with  a 
detailed  knowledge  of  the  radars,  gained  from  the  documents 
referenced  above  and  recent  data  from  Service  data  systems,  the 
three  most  recent  radars  were  selected  to  construct  the  reference 
line.  The  judgment  to  use  these  three  radars  was  made  because 
they  are  considered  to  be  successful  developments  with  a  reasonable 
balance  of  R&M  factors  and  because  they  were  essentially  new  designs. 
In  order  to  make  the  tool  more  useful  throughout  the  range  of 
potential  applications,  the  line  was  forced  through  the  origin 
(0,0).  The  resulting  reference  line  is  shown  on  Fig.  B-29.  The 
next  step  tests  the  utility  of  the  chart.  Again,  the  purpose  of 
this  tool  is  to  raise  "red  flags"  when  further  investigation  is 
warranted. 

The  remaining  four  radars  (with  Table  B-2  as  a  reference) 
were  used  to  test  the  utility  of  the  approach.  The  APO-113 
radar  for  the  F-111A  and  F-111E  falls  considerably  to  the  right 
of  the  reference  line.  That  is,  if  the  line  represents  a  reason¬ 
able  amount  of  time  from  go-ahead  to  the  first  production  delivery 
for  a  given  complexity  (parts  count),  the  APO-113  exceeds  the 
time  roughly  by  a  factor  of  two  (36  months  actual  versus  17 
months  using  the  reference  line).  The  GE  study,  along  with 
historical  trends  in  fire  control  radar  development  provides 
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insight  which  indicates  that  the  reference  line  is  useful  for 
our  intended  purpose.  The  APQ-113  program  was  a  watershed 
program  in  many  ways:  (1)  its  design  was  among  the  first  to 
depend  heavily  on  solid-state  components  (new  technology),  (2) 
the  program  had  a  very  strong  reliability  requirement  that  was 
enforced,  (3)  many  new/evolving  design,  parts  selection  and 
parts  screening  techniques  were  used  (e.g.,  1,413  hours  of  ROT, 
7,000  hours  of  RAT,  89  percent  high  reliability  parts  selected, 
a  very  detailed  design  integration  to  reduce  parts  count,  etc.). 
The  combination  of  these  factors  indicates  that  the  APQ-113  was 
first  to  do  many  things — a  learning  process  for  the  Air  Force 
and  the  contractor.  Although  the  time  to  first  production  article 
appears  excessive  (compared  to  the  reference  line)  a  great  deal 
was  learned  from  the  program  which  has  been  refined  and  used  in 
all  radar  development  programs  since. 

If  a  new  program  was  being  reviewed  today  and  it  fell  in  the 
same  area  of  the  chart  as  the  APQ-113,  one  would  want  to  know  why 
(e.g.,  is  new  technology  being  applied?,  are  new  untried  design 
approaches  being  used?,  etc.).  If  the  potential  benefits  were 
judged  to  be  worth  the  extra  time  (i.e.,  watershed  for  future 
developments),  the  program  could  be  approved  or  other  alternatives 
like  off-line  maturing  could  be  explored. 

The  next  two  radars  used  to  test  the  approach  are  the  APQ-114 
and  APQ-144.  Both  of  these  radars  are  direct  derivatives  of  the 
APQ-113  as  shown  in  Fig.  B-30.  Considering  the  concurrency  of  the 
APQ-114  and  APQ-113  programs  and  the  cross-flow  into  the  APQ-144 
program,  the  113  &  144  points  tend  to  add  validity  to  the  proposed 
review  approach. 

Finally,  the  APQ-120  program  was  used  to  test  the  validity 
of  the  approach.  The  APQ-120  program  is  a  good  case  history  of  a 
fast  track  (need  it  as  soon  as  possible)  program.  The  radar  was 
needed  to  meet  Southeast  Asia  needs.  It  was  developed  quickly 
without  substantial  design  iteration  and  virtually  none  of  the 
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FIGURE  B-30.  Design  Inheritance,  APQ-113 


testing  and  parts  selection  techniques  used  for  the  APQ-113,  and 
the  three  new  radars  used  to  construct  the  reference  line  in 
Pig.  B-29.  It  was  also  produced  at  a  very  high  rate,  as 
shown  on  Fig.  B-31.  (The  lower  the  line  on  Fig.  B-31,  the  faster 
the  rate  of  production.)  In  the  APQ-120  program,  600  units  were 
delivered  in  36  months.  As  might  have  been  predicted  from  the 
reference  line  and  other  known  information  on  the  treatment  of  R&M, 
the  APQ-120  has  suffered  from  its  heritage  with  low  reliability,  and 
has  had  very  little  opportunity  to  grow  its  reliability  short  of 
a  total  redesign. 

The  tool  discussed  above  is  not  a  substitute  for  a  program 
structuring  process  where  knowledgeable  contractor,  Service 
acquisition  specialists  and  user  personnel  interact  to  develop  a 
balanced  acquisition  plan.  In  fact,  as  discussed  above,  the 
three  programs  used  to  construct  the  reference  line  were  selected 
because  they  were  balanced.  That  is,  they  contained  a  reasonable 
mixture  of  proven  techniques  that  are  known  to  enhance  R&M 
characteristics.  Some  of  those  techniques  are  listed  in  Table  B-3. 

A  more  detailed  discussion  of  this  approach  is  contained  in  Appendix 
B  and  in  the  Navy  "New  Look  Approach." 

TABLE  B-3.  Front-End  Task/Techniques  for  R&M 

•  REASONABLE  YET  CHALLENGING  R&M  REQUIREMENTS 

•  PLANNED  DESIGN  ITERATION 

•  PARTS  SELECTION  PROGRAM 

•  ENVIRONMENTAL  TESTING  ( RDT,  RAT,  ETC.) 

e  OPERATIONAL  TESTING 

•  ENVIRONMENTAL  STRESS  SCREENING 

•  PLANNED  MATURATION  PROGRAM 
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APPENDIX  C 


ELECTRONIC  WEAPON  SYSTEMS  ANALYSIS 


This  Appendix  discusses  the  analysis  of  the  structuring  and 
execution  of  reliability  and  maintainability  programs  as  observed 
across  electronic  weapon  systems  programs  reviewed.  It  also  con¬ 
sidered  other  information  that  was  presented  or  obtained  during 
the  course  of  the  study.  Observations  and  analyses  contained  in 
this  section  are  primarily  based  on  the  review  of  the  radar  case 
study  reports  for  the  F-15  (APG-63),  F-16  (APG-66),  F/A-18  (APG-65) , 
FIREFINDER  (TPQ-36/37),  Lightweight  Doppler  Navigation  System 
( APN-128 )  and  AEGIS  radar  (AN/SPY-1A) .  These  radar  developments 
cover  a  total  time  period  of  about  15  years  and  represent  the 
best  efforts  of  all  three  services,  i.e..  Army:  FIREFINDER,  Navy: 
AEGIS  and  F/A-18  radars.  Air  Force:  F-15  and  F-16  radars.  The 
purpose,  performance,  and  complexity  vary  widely  for  the  systems 
(in  terms  of  complexity  the  number  of  component  parts  involved 
range  from  less  than  5,000  parts  to  over  500,000  parts). 

For  this  analysis,  the  R&M  program  activities  were  divided  into 
five  major  categories:  contracting,  management,  design,  manufac¬ 
turing,  and  test  and  evaluation.  These  five  categories  were 
divided  further  into  26  elements.  A  Program  Review  Document, 
developed  in  this  study,  was  used  to  provide  a  mutual  understanding 
of  the  26  elements  and  to  communicate  the  scope  and  content  of 
the  elements  to  personnel  in  the  government  program  offices  and 
contractors  involved  in  the  individual  acquisition  programs 
that  were  examined. 

As  a  result  the  observations  and  findings  associated  with 
these  electronic  programs  are  considered  valid  for  future  programs 
for  complex  electronic  systems.  To  a  les'ser  extent,  these  results 
may  also  be  appropriate  for  newer  generation  developments  such  as 
electrooptical  systems  but  no  effort  has  been  taken  to  verify  t,  at 
claim. 
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A.  CONTRACTING 

This  section  addresses  the  seven  contracting  elements  identified 
and  defined  in  IDA  Record  Document  D-26,  R&M  Program  Review  Elements. 

1 .  R&M  Requirements 

The  procuring  activities  for  each  of  the  radars  appeared 
to  appreciate  the  necessity  for  firm  R&M  contract  requirements 
with  some  form  of  verification.  They  avoided  the  procedure  used 
a  number  of  years  ago  where  "goals"  were  specified  and  R&M  did 
not  receive  proper  attention.  R&M  values  were  stated  in  slightly 
different  ways  and  varied  as  lessons  were  learned  from  previous 
contracts.  Mean-time-between-f ailures  (MTBF),  based  on  radar 
operating  time,  is  used  as  a  contractual  measure  of  hardware 
reliability,  whereas  mean-fl ight-hours-between-fai lures  (MFHBF ) 
is  used  to  reflect  service  operational  reliability  performance. 
Although  the  definition  of  "reliability"  in  most  military  standards 
(MIL-STD-721 ,  for  example)  includes  a  "probability  of  success," 
this  is  seldom  used  any  more  since  it  is  difficult  to  measure. 
Maintainability,  however,  is  different  in  that  probabilities  or 
percentages  are  useful  to  describe  fault  isolation/BIT  charac¬ 
teristics  (i.e.,  95  percent  fault  detection).  Maintainability 
sometimes  uses  time  and  percentages  to  describe  requirements  (e.g., 
120  minutes  max  for  90  percent  of  all  maintenance  actions). 
Mean-time-to-repair  (MTTR)  and  maintenance  man-hours/flight  hour 
(MMH/FH)  are  widely  used  in  specifying  and  measuring  maintainability. 

Although  the  methods  used  by  the  Services  to  define  R&M  re¬ 
quirements  were  substantially  different,  each  method  worked  and 
each  program  has  been  successful.  Methods  used  reflected  the 
popular  notions  at  the  time  the  request  for  proposal  (RFP)  was 
being  prepared.  Figure  C-l  summarizes  the  salient  R&M  require¬ 
ments  associated  with  the  radar  cases  studied: 
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FIGURE  C-l.  R&M  Requirements  Summarized 
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a.  R&M  Contract  Goals  versus  Requirements 
It  is  widely  recognized  today  that  when  R&M  contract 
requirements  are  specified  as  goals  instead  of  verifiable  require¬ 
ments,  they  do  not  receive  proper  attention  by  the  contractor. 

When  R&M  values  were  specified  as  goals,  as  was  the  procedure  a 
number  of  years  ago,  technical  performance  received  a  higher 
priority  in  the  minds  of  management.  Engineers  tended  to  design 
their  equipment  to  pass  the  test  section  of  a  specification. 

Goals  without  a  verification  test  received  a  low  priority  in  the 
design  process.  Not  only  must  R&M  be  stated  as  a  requirement, 
but  verification  tests  need  to  be  identified  and  growth  expecta¬ 
tion  specified.  Each  of  the  six  radar  cases  studied  contained 
firm  R&M  requirements  with  preplanned  verification  tests  which 
provided  for  growth.  In  one  case,  the  FIREFINDER  (AN/TPQ-36), 

MTBF  "goals"  of  400  hours  (four  times  as  high  as  the  required 
100-hour  MTBF)  were  also  included  in  the  contract  (Ref.  FIRE- 
FINDER  Case  Study,  IDA  Record  Document  D-24,  p.  9).  Experience 
would  indicate  that  the  400-hour  goal  value  will  not  be  an  effec¬ 
tive  method  of  obtaining  improved  equipment  R&M.  Similarly, 
the  F-15  RFP  initially  required  a  100-hour  avionic  subsystem 
MTBF  with  a  goal  (objective)  of  150  hours  (Ref.  F-15  Case 
Study,  IDA  Record  Document  D-19,  p.  IIA-7). 

The  FIREFINDER  case  study  provides  a  good  example  of  Goals 
versus  Requirements.  The  TPQ-37  Advanced  Development  Contract  con¬ 
tained  only  a  250-hour  MTBF  Goal  and  it  appears  that  the  contractor 
placed  the  major  emphasis  on  performance  (Ref.  FIREFINDER  Case  Study, 
p.  58).  When  the  TPQ-37  system  went  from  Advanced  Development  to 
Low  Rate  Initial  Production  (LRIP),  a  firm  90-hour  MTBF  requirement 
was  added  to  the  contract,  along  with  an  incentivized  Reliability 
Demonstration  Test.  The  measured  MTBF  went  from  33  hours  for 
Advanced  Development  configuration  to  115  hours  for  the  LRIP 
radars.  Many  things  contributed  to  the  Reliability  improvement; 
however,  it  is  believed  that  the  management  emphasis  placed  on 
meeting  the  Reliability  requirements  during  the  incentivized  demon¬ 
stration  test  was  a  major  factor. 
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b.  Developing  R&M  Requirements 


Properly  defined,  realistic,  but  challenging  R&M  require¬ 
ments  are  fundamental  to  successful  R&M  development.  Program 
R&M  requirements  establish  the  basis  for  determining  resource 
needs  and  the  R&M  approach  during  design,  development  and  produc¬ 
tion.  The  successful  cases  studied  emphasized  the  word 
"realistic."  An  unrealistically  high  requirement  can  result  in 
excessive  program  costs  and  program  disruptions  when  the  figure 
cannot  be  met.  Too  low  a  requirement  may  result  in  too  little 
R&M  emphasis  and  the  loss  of  an  opportunity  to  get  significantly 
high*  r  R&M  at  essentially  no  increase  in  acquisition  cost. 

Program  R&M  requirements  should  ensure  that  the  program 
produces  a  system/equipment  which  is  operationally  adequate  and 
logistically  supportable.  Contract  requirements  need  not  be 
specified  in  operational  terminology,  but  they  must  be  selected 
such  that  achievement  of  the  contract  requirements  ensures  that 
the  operational  and  logistics  support  requirements  will  be  met. 

Many  of  the  successful  cases  studied  followed  the  fore¬ 
going  defined  approach.  In  the  F-16  radar  case,  comparability 
studies  were  done  using  operational  data  factored  for  improve¬ 
ments  expected  from  technology  and  better  design.  Operational 
experience  of  previous  radar  systems  was  compared  to  MIL-STD-781 
test  results  of  the  same  systems  to  determine  what  the  con¬ 
tractual  test  requirements  should  be  to  provide  a  high  probability 
of  achieving  the  projected  field  performance.  Significantly, 
the  F-16  radar  has  surpassed  field  projections. 

Realistic  yet  challenging  R&M  requirements  can  be  defined 
using  analytical  techniques  and  past  operational  experience. 
Analytical  techniques  are  available  to  establish  the  relation¬ 
ships  between  R&M  and  operational  impacts,  and  logisitics 
supportability.  Operational  experience  is  available  to  establish 
baselines  which  can  be  useful  in  developing  rational  engineering 
projections  of  what  can  be  realistically  achieved. 
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The  establishment  of  testability  requirements,  particularly 
in  the  area  of  built-in-test,  has  evolved  slowly  over  the  past 
fifteen  years.  Most  programs  have  followed  the  lead  of  their 
predecessors  and  have  made  only  small  changes  and  improvements  in 
capability.  There  have  been  very  large  increases  in  computer 
power  (currently  210K  16-Bit  words  for  the  F/A-18)  with  advanced 
revisions  of  the  F-15  growing  to  8.2  megabits  total  capacity. 

This  should  mean  better  fault  isolation  where  LRUs  or.  modules  can 
be  redesigned  to  capitalize  on  the  increased  memory,  especially 
in  digital  circuits.  The  ability  of  BIT  to  detect  and  isolate 
failures  varies  with  the  type  of  LRU/SRA  and  module  with  analog 
processors  and  receivers  being  more  difficult  than  digital  or 
signal  processors.  Fault  detection  requirements  indicate  some 
increase  in  the  level  of  detection  from  95  percent  for  the  F-15, 
and  95  percent  for  the  F-16  to  98  percent  for  the  F/A-18. 

AEGIS  is  indicating  100  percent  fault  detection  coverage, 
monitoring  operability  test,  and  a  periodic  test.  For  the  FIRE- 
FINDER,  the  requirements  are: 

•  Fault  Isolation  and  Repair  on  Site 

-  Organizational  -  90  percent  of  repairable  faults 

-  Direct  support  -  10  percent  of  repairable  faults 

•  Automatic  Fault  Isolation  (BIT)  to: 

-  1  unit  for  75  percent  of  failures 

-  2  units  or  less  for  90  percent  of  failures 

-  8  units  or  less  for  98  percent  of  failures 

(Ref.  FIREFINDER  Case  Study,  p.  61). 
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2.  Mission  Profiles 

Mission  profile  development  should  begin  in  the  earliest 
program  phase  and  continue  to  some  degree  throughout  the  program. 

If  mission  profiles  are  not  defined  early  in  the  program,  then 
critical  design  decisions  may  not  consider  the  expected  operational 
stresses.  Each  of  the  programs  studied  approached  the  derivation 
and  use  of  mission  profiles  in  different  ways,  being  influenced 
by  the  practices  in  use  at  the  time  and  by  the  Military  Service 
involved.  The  following  is  a  summary: 

•  In  the  1970  time  period  when  the  F-15  program  was 

initiated,  there  was  an  emphasis  on  the  Point  Inter¬ 
cept  Mission  and  making  the  F-15  an  Air  Superiority 
Fighter  (Ref.  F-15  Case  Study,  p.  IA-2).  Modifications 
for  other  mission  capabilities  that  added  cost,  weight, 
or  complications  were  not  allowed.  The  byword  was 
"not  a  pound  for  air-to  ground"  capabilities.  At  one 
time,  it  was  planned  to  fly  a  number  of  dedicated 
point  intercept  missions  to  demonstrate  R&M ,  but  this 
idea  was  later  dropped.  After  a  review  of  the  various 
F-15  missions,  it  was  decided  that  Test  Plan  III, 

Level  F  of  MIL-STD-781B  would  be  used  as  the  basis  of 
reliability  demonstration  tests.  Test  Level  F  called 
for  testing  from  -54°C  to  +71°C  with  sine  vibration 
between  20  and  60  Hz  with  2.2  g's  ±  10%  oeak  accel¬ 
eration.  During  initial  design,  it  was  found  that 
additional  hardware  (heaters,  control  circuits,  etc.) 
would  be  required  for  consistent  repeated  turn-on  at 
-54°C  (Ref.  F-15  Case  Study,  p.  IC-4).  The  test  con¬ 
dition  was  modified  to  radar  turn-on  at  -40°C  following 
stabilization  at  -54°C  in  the  off  mode  (Ref.  FIREFINDER 
Case  Study,  p.  IIE-19). 
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•  In  1973,  when  the  FIREFINDER  program  was  initiated 
by  the  Army,  the  environmental  requirements  for 
weapon- locating  radars  were  well-defined  and  the 
mission  profiles  established.  The  mission  profile 
for  the  TPQ-36  was:  (a)  operational  time  10  days 
at  24  hrs/day,  (b)  Travel  and  Maintenance  mode 

21  hrs/day,  (c)  emplacement/displacements 
3  hrs/day  (d)  scheduled  and  unscheduled  main¬ 
tenance  time  1  hr/day.  The  system  had  to  operate 
under  worldwide  environments  per  Army  Regulation 
70-38  (up  to  125 °F  ambient  conditions)  in  addition 
to  rain,  and  transportation  over  rough  terrain  (Ref. 
FIREFINDER  Case  Study,  p.  65). 

•  By  1974,  when  the  F-16  radar  procurement  was  initiated, 
it  was  recognized  that  the  ten  minutes  of  sine 
vibration  at  2.2  g's  peak  acceleration  was  effective 
mostly  as  a  means  to  dislodge  loose  solder.  Random 
vibration  appeared  more  representative  of  mission 
conditions.  As  a  result,  reliability  tests  were 
conducted  with  random  vibration  at  2.96  g's  RMS 
between  20  and  2000  Hz.  Level  F  was  specified  with 
-40°C  turn-on  (similar  to  the  F-15). 

•  In  1976,  when  the  F/A-18  program  was  initiated,  the 
Navy's  "new  look"  program  emphasized  the  importance 
of  mission  profile  definition  in  all  areas  of  design 
and  test  endeavors.  An  Operational  Mission  Environ¬ 
ment  (OME)  was  derived,  based  on  the  frequency 

of  occurrence  for  each  mission  established  for  Navy 
Fighter,  Navy  Light  Attack,  and  Marine  Fighter/Attack 
Squadrons,  as  well  as  ship/shore  and  combat/training 
sortie  ratios.  The  resulting  OME  formed  the  basis 
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for  establishing  expected  flight  load,  vibration, 
temperature,  altitude,  humidity,  acoustic,  salt,  and 
dust  conditions.  The  OME  concept  permeated  and  in¬ 
fluenced  thinking  in  all  design  areas  (Ref.  F-18  Case 
Study,  p.  70-71).  Since  the  Reliability  Development 
(TAAF)  tests  were  established  long  before  the  OME 
was  completely  developed,  these  tests  were  a  mixture 
of  different  test  philosophies.  Test  Level  F  (-54° 
stabilization,  -40°C  turn-on  and  operation  up  to  +71°C) 
and  various  vibration  exposures  were  used.  These 
were:  (a)  12.6  g's  RMS  at  50  to  2000  Hz  for  non-gun¬ 
fire  simulation,  (b)  21.6  g's  RMS  for  endurance,  (c) 
37.2  g's  RMS  at  500  to  2000  Hz  for  gunfire,  (d)  sine 
vibration  at  0.2  to  4.6  g's  in  the  5  to  50  Hz  range  for 
non-gunfire,  and  (e)  10  to  18  g's  in  the  50  to  500  Hz 
range  for  gunfire. 
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Life  Profiles 


Detail  design  actions  must  also  address  the  stresses  that  are 
projected  for  the  system's  life  profile.  The  non-mission  portion 
of  a  system's  life  can  include  design  requirements  that  are  not  con¬ 
tained  in  the  defined  mission  profile.  None  of  the  case  studies 
specifically  addressed  this  subject  but  it  is  known  that  each 
program  had  requirements  to  provide  for  storage,  packaging,  trans¬ 
portation  and  handling  effects  on  the  radar  equipment. 

The  following  is  a  case  history  from  the  F-15  that  may  be  re¬ 
presentative  of  most  programs.  Considerable  effort  has  been  ex¬ 
pended  by  Hughes  to  provide  insulated  shipping  containers  for  trans¬ 
porting  radar  LRUs  from  Hughes  in  Los  Angeles  to  MCAIR  in  St.  Louis 
and  other  worldwide  destinations.  These  containers  appeared 
satisfactory  for  several  years.  In  1978  MCAIR  started  observing 
damage  to  the  outer  perimeter  ring  of  antennas  and  thought  that 
they  were  causing  this  on  aircraft  installation.  Further  in¬ 
vestigation  and  test  revealed  that  the  antenna  attaching  clamps 
were  not  securely  holding  the  antenna  during  shipment.  Redesign 
to  the  interior  of  the  shipping  package  solved  the  problem.  As 
an  added  precaution,  accelerometers  were  installed  in  each  type 
of  radar  LRU  shipping  container  to  ascertain  if  the  shock  during 
shipment  could  cause  damage  to  the  LRUs. 

4.  R&M  Failure  Definition 


R&M  requirements  have  complete  meaning  only  if  established 
in  a  defined  context.  Variations  of  failure  definition,  time, 
critical  failures,  etc.,  result  in  wide  variations  in  the  meaning 
of  R&M  parameters  and  inadequate  communication  between  govern¬ 
ment  and  contractors.  Agreements  and  definitions  must  be  in 
writing  and  established  at  an  appropriate  point  in  the  contract. 
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The  hardware  attribute  commonly  referred  to  as  "reliability" 
is  generally  recognized  as  one  of  several  contributors  to  the 
frequency  of  field  maintenance.  Other  external  contributors 
include  adequacy  of  test  equipment#  manuals#  training#  and  the 
attitude  and  skill  levels  of  equipment  operators  and  repair 
personnel.  There  is  a  tendency  for  users  to  compare  frequency  of 
maintenance  to  inherent  "reliability,"  be  unhappy  with  the 
results#  and  blame  failure  definitions  and  ground  rules. 

The  three  airplane  radar  contracts,  F-15,  F-16,  and  F/A-18 
were  conducted  under  MIL-STD-781B ,  "Reliability  Tests:  Exponen¬ 
tial  Distribution,"  which  in  paragraph  5.5.1  defines  failure 
categories  (Ref.  F-15  Case  Study,  p.  II  A-3;  F/A-18,  p.  77). 

According  to  the  standard,  all  failures  are  relevant  and 
chargeable  unless  and  until  determined  to  be  non-relevant  or 
non-chargeable ,  or  both,  by  the  procuring  activity.  As  a  rule, 
contractors  collect  failure  data,  analyze  the  occurrence  and 
propose  a  relevant  or  non-relevant  classification.  The  burden  of 
proof  is  the  contractor's  responsibility.  The  final  judgment  is 
rendered  by  the  procuring  activities.  Failure  category  definitions 
are  expanded  upon  in  Reliability  Test  Plans,  which  are  required 
to  be  submitted  after  contract  go-ahead.  Procuring  activities 
and  contractors  generally  concurred  in  the  following: 

•  Installation  damage,  mishandling,  test  equipment 
failures  are  non-relevant. 

•  Secondary  CFE  failures  are  non-chargeable  if  a  primary 
(causative)  CFE  or  GFE  failure  has  been  charged. 

•  Pattern  failures  require  corrective  action. 

In  the  F/A-18,  a  ground  rule  used  during  FSD  included 
provisions  for  classifying  a  failure  as  non-relevant  if  a  fix  had 
been  identified  prior  to  the  field  occurrence.  (This  approach 
was  a  compromise  between  counting  all  failure  occurrences  until 
the  fix  was  implemented  and  not  counting  repeats  of  known  problems, 
(Ref.  F/A-18  Case  Study,  p.  76).  One  contractor  believes  that  this 
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method  projects  a  realistic  trend  as  to  what  can  be  expected  in 
a  mature  system  in  the  field.  This  is  definitely  somewhat 
optimistic  for  early  field  results  but  will  be  closer  after 
early  support  problems  are  resolved.  Another  position  is  that 
adjustment  of  demonstration  test  results  on  the  basis  of  corrective 
actions  not  yet  incorporated  is  an  appropriate  procedure,  given 
an  adequate  understanding  of  the  failure  event  and  the  effective¬ 
ness  of  the  associated  corrective  action.  Based  on  AMSAA  Technical 
Report  375,  June  1982,  and  a  presentation  by  Dr.  Larry  Crow,  U.S. 
Army  Materiel  System  Analysis  Activity,  Army  experience  indicates 
that  this  procedure  is  optimistic  and  generates  projected  values 
which  on  an  average  overestimate  system  reliability. 

The  FIREFINDER  failure  definitions  are: 

•  Relevant  Failure  (chargeable) 

-  Reduces  System  Performance  below  Specified  Levels 

-  Caused  by  Design  or  Manufacturing  Defects  or  Physical 
Deterioration 

•  Non-Relevant  Failure  ( Non-Chargeable ) 

-  Damage  from  Improper  Installation,  Mishandling,  or  Abuse 

-  Failure  due  to  Error  in  Test  Procedures 

-  Failure  due  to  Externally  Induced  Overstress 

-  Operator  Errors 

-  Secondary  Failures 

-  Failure  of  GFE  Items 

-  Redundant  Items 
(Ref.  FIREFINDER  p.  69). 

Since  the  AEGIS  System  design  involved  the  concept  of  multiple 
redundancies  in  all  principal  functions  and  can  accept  certain 
malfunctions/failures  as  long  as  acceptable  performance  is  main¬ 
tained,  their  failure  definitions  are  different  from  those  of  an 
airborne  radar.  AEGIS  definitions  are  listed  in  the  following, 
with  Fig.  C-2  depicting  graphically  performance  profile  charac¬ 
teristics. 
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Major/Critical  Events 

•  Events  that  reduce  performance  below  specified 
thresholds 

•  Restoration  is  manual  and  completed  in  minutes  or 
hours 

•  Based  on  72-hour  mission  profile. 


Interrupt  Events 

•  Events  that  temporarily  reduce  performance  below 
specified  thresholds 

•  Restoration  is  automatic  and  completed  in  seconds 

•  Based  on  72-hour  mission  profile. 
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FIGURE  C-2.  AEGIS  Performance  Profile  Characteristics 
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5.  Incentives 

R&M  contract  incentives  can  aid  in  developing  reliable  and 
maintainable  systems  by  focusing  contractor  management  attention 
on  actions  that  improve  and  ultimately  meet  specified  requirements. 

The  F-16,  F/A-18,  and  LDNS  programs  contained  incentives  that 
were  successfully  implemented  to  the  contractor's  and  the  govern¬ 
ment's  benefit. 

The  RIW  (Reliability  Improvement  Warranty)  played  an  important 
role  in  two  programs.  These  were: 

(1).  Lightweight  Doppler  Navigation  System  (LDNS):  the  con¬ 
tractor  warranted  that  the  LDNS  units  furnished  were  free  of 
defects  in  material,  workmanship  and  design,  and  would  operate 
in  the  intended  environment  for  the  specified  warranty  period. 

The  contract  provides  for  renewal  of  the  warranty. 

Any  unit  that  failed  to  meet  the  warranty  and  was  returned  to  the 
contractor  was  to  be  repaired  or  replaced  at  the  contractor's  sole 
option  and  expense.  The  contractor  is  not  obligated  to  perform 
cosmetic  repairs.  Repaired  or  replaced  items  were  to  be  tested 
against  a  specified  Acceptance  Test  Procedure.  The  government 
witnessed  test  activity  and  reviewed  the  documented  results. 

For  purposes  of  the  warranty,  the  Initial  Anniversary  Date  ( IAD) 
was  the  date  of  successful  completion  of  DT  III  PVT-G  (Production 
Validation  Testing-Government)  Testing.  This  date  was  used  to 
establish  reporting  and  adjustment  periods  for  the  warranty. 

For  all  Low  Rate  Initial  Production  units,  the  initial  warranty 
period  started  when  the  government  accepted  a  unit  and  extended  48 
months  after  the  IAD.  The  contractor  and  contracting  officer 
negotiated  the  price  for  any  renewal  of  the  warranty  period. 

The  contractor  does  not  pay  for  repair/replacement  of  units  for 
nonconformance,  loss  or  damage  due  to: 

(1)  Non-LDNS  induced  fire  or  explosion 

(2)  Submersion 

(3)  Aircraft  crash 
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(4)  Enemy  action- 

(5)  Natural  disaster,  or 

(6)  Accidental  or  willful  mistreatment. 

The  exclusions  did  not  apply  at  contractor-controlled  locations 
or  if  the  LDNS  caused  one  or  more  of  the  above  events.  Clear  and 
convincing  evidence  was  to  accompany  the  contractor's  claim  from 
relief  from  Warranty  Obligation  for  any  of  the  above  listed 
exclusions. 

The  contractor  was  to  repair  or  replace  any  defective  unit  in 
accordance  with  the  terms  of  the  warranty.  The  Contractor  was  not 
liable  for  special  or  consequential  damages  (Ref.  LDNS,  IDA  Record 
Document  D-23  pp.  65/2-3,  4). 

Additional  cost  to  the  government  for  the  RIW  provision  is 
priced  with  each  batch  of  units  shipped.  These  costs  generally 
represent  between  10  and  15  percent  of  the  procurement  cost. 

(2)  F-16  Radar  System:  In  an  attempt  to  motivate  the  contrac¬ 
tor  further  after  he  won  the  contract  award  in  1974,  the  Air  Force 
included  in  the  contract  an  option  to  exercise  RIW  provisions. 

In  1977,  a  contract  was  subsequently  signed  with  GD  for  RIW 
coverage  of  five  (out  of  seven)  radar  LRUs  tor  the  U.S.  and  the 
European  Participating  Governments  (EPG)  in  the  Multinational 
Fighter  Program.  The  warranty  applied  to  all  units  installed  in 
the  first  250  USAF  and  the  first  192  EPG  production  aircraft  and 
to  spares  procured  for  support  of  those  aircraft. 

The  LRUs  selected  were: 

e  Transmitter 
e  Signal  Processor 
e  Computer 
e  Receiver 
e  Antenna. 

The  radar  control  panel  and  rack,  cable  and  waveguide  assembly 
LRUs  were  excluded  from  RIW.  Figure  C-3  lists  the  major  features 
of  the  RIW  program. 


Characteristic _ _ _  _ _ Description 


Incentive  award  fees  were  effectively  utilized  on  two  programs 

These  were: 

(1)  F/A-18  Radar  Program:  As  part  of  the  Navy's  "New  Look"  in 
R&M,  an  incentive  award  fee  was  issued  as  part  of  the  basic 
contract  to  provide  MCAIR  an  opportunity  to  gain  awards  based  on 
demonstrated  aircraft  performance  in  the  areas  of  R&M.  These 
award  fees  were  then  structured  to  allow  major  suppliers  to 
participate  in  the  Navy's  "New  Look”  R&M  incentive. 

The  reliability  features  of  the  radar  to  be  demonstrated 
were  MTBF  and  MFHBF.  The  maintainability  features  were  MMH/FH 
(O- Level  Unscheduled),  DMMH/FH  (O&I  Total)  and  MFHBMA  (0-Level). 
These  requirements  were  selected  to  be  demonstrated  during  the 
production  reliability  test,  the  1200FH,  2500FH,  and  9000FH  periods 
The  incentive  award  fee  was  structured  to  provide  60  percent  of  the 
total  award  pool  to  reliability  and  40  percent  to  maintainability 
(Ref.  F/A-18  Case  Study,  p.  78). 

Financial  incentives  were  provided  contractually  in  the  F/A-18 
radar  program  as  shown  in  Fig.  C-4. 

(2)  FIREFINDER  Radar  Program:  The  Reliability  Improvement 
Program  (RIP)  for  the  TPQ-37  created  an  incentive  in  dollars 
for  the  contractor  to  exceed  a  90-hr  MTBF  during  a  government- 
funded  demonstration.  The  value  of  the  incentive  was  $5.5 
million,  less  the  cost  of  the  RIP.  The  objective  was  to  achieve 
17  or  fewer  failures  in  1500  hours  of  testing.  The  cost  of  the 
RIP  was  $4.8  million.  Thirteen  failures  were  experienced  and 
the  contractor  earned  approximately  90  percent  of  the  incentive 
or  $508K.  The  incentive  schedule  was  as  follows: 

FAILURES  IN  1500  HOURS  $  INCENTIVE  ACTUAL 


F/A-18  APG-65  RADAR 
R&M  INCENTIVE  AWARD  FEE  STRUCTURE 


6.  Source  Selection  Criteria 


In  a  recent  LMI  review  of  20  major  weapon  systems  programs, 
it  was  evident  that  government  source  selection  criteria  had  not 
consistently  placed  adequate  priority  on  R&M  (Fig.  C-5). 


TOTAL  NUMBER 

OF  PROGRAMS 
REVIEWED 

NUMBER  WITH 

R&M  IN  SOURCE 
SELECTION 

PERCENT 

ARMY 

6 

2 

33% 

NAVY 

6 

0 

0% 

AIR  FORCE 

8 

4 

50% 

TOTAL 

20 

6 

30% 

(LMI  Working  Note:  "A  Documentation  of  DoD  Strategies  for 
Acquiring  Weapon  System  Reliability  and  Support"  Dec.  1982 
pp.  2-2,  2-3,  and  2-4) 


FIGURE  C-5.  R&M  is  not  Consistently  Used  in  Source  Selection 


Programs  reviewed  in  the  LMI  study  consisted  of  6  Army  programs 
6  Navy  Programs,  and  8  Air  Force  programs.  Army  programs  included 
the  M-l  Tank,  Patriot  Air  Defense,  Bradley  M2/M3  Infantry  Fighting 
Vehicle,  Apache  AH-64  Helicopter,  Blackhawk  UH-6A  Utility  Tactical 
Transport,  and  DIVAD  Air  Defense  Systems.  Navy  systems  consisted 
of  the  F/A-18  Fight<er  Attack  Aircraft,  Tomahawk  Submarine  Launched 
Cruise  Missile,  F-14A  First-Line  Fighter,  S-3A  Viking  Anti-Submarine 
Warfare  Aircraft,  MK-86  Gun  Fire  Control  System,  AEGIS  Shipboard 
Air  Defense  and  Phalanx  Close-In  Weapon  System. 

Air  Force  Programs  were  the  B-1B  Bomber,  NGT  Trainer,  KC-10 
Tanker,  A-10  Close  Support  Fighter,  F-16  Lightweight  Fighter, 

ALCM  Air-Launched  Cruise  Missile,  AWACS  Airborne  Warning  and  Control 
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and  GPS  NAVSTAR  Global  Positioning  System.  Of  the  weapon  systems 
reviewed  by  LMI,  individual  case  studies  were  done  for  the  F-16 
Radar,  the  F/A-18  Radar,  and  the  Blackhawk  T700  engine. 

As  reported  in  the  T700  case  study,  the  inclusion  of  Reliabil¬ 
ity  and  Maintainability  in  the  Source  Selection  criteria  left  no 
doubt  that  the  Army  was  serious  about  R&M  Requirements  (Ref.  T700 
Case  Study,  p.  IIA-75).  Likewise,  the  F-16  Radar  case  study 
shows  that  Reliability  and  Maintainability  were  key  factors  in 
the  RFP  (Ref.  F-16  Radar,  p.  IIA-14). 

Although  the  LMI  study  reported  that  R&M  was  not  used  in 
source  selection  for  the  F/A-18  program,  the  individual  case 
study  for  the  F/A-18  Radar  reported  that  the  importance  of  R&M  in 
source  selection  for  the  radar  was  clearly  established  through 
briefings,  request  for  proposal  instructions  and  hard  specification 
requirements.  In  addition,  R&M  evaluation  was  conducted  in  all 
key  proposal  areas  including  Design,  Manufacturing/Production 
Plan,  Management  and  contractual,  not  just  in  the  R&M  proposal 
volumes  (Ref.  F/A-18  Radar  Case  Study,  p.  87). 

From  the  case  studies  analyzed,  it  is  evident  that  when  the 
hardware  procurers  placed  heavy  emphasis  on  R&M  in  the  source 
selection  process,  then  the  contractors  and  suppliers  in  turn 
reflected  that  relative  importance. 
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' .  Life-Cycle  Cost  Considerations 

A  system's  life-cycle  cost  is  a  function  of  R&M  variables  and 
can  be  reduced  by  selecting  and  achieving  certain  R&M  requirements. 
During  design  trade  studies,  LCC  was  used  as  one  factor  to  evaluate 
alternative  designs.  LCC  efforts  were  similar  among  the  contractors 
studied.  Some  examples  are: 

a.  F-15  -  Life-Cycle  Cost  analyses  were  conducted  in  the 

initial  design  formulation  studies  and  during  trade 
study  activities.  Figure  C-6,  "BIT  Mechanization  Trade 
Study,"  lists  costs  as  well  as  R&  nd  other  factors 
considered  (Ref.  F-15  Case  Study  i  IIC-7).  Throughout 
the  development  history  of  the  r  ar  (1972-1983)  there 
were  numerous  changes.  The  ten  ^sentative  VECP 

changes  (see  pp.  IIC-9  through  21  c  the  F-15  Case  Study 
Report)  were  LCC  evaluated  by  Hughes  and  checked  by 
MCAIR  before  incorporation  into  VECP  and  submittal  to 
the  Air  Force. 

b.  F-16  -  Westinghouse  conducted  LCC  analyses  and  submitted 
them  to  General  Dynamics  rnd  the  Air  Force  using  cost 
models  specifically  designed  to  measure  LCC.  A  major 
concern  in  the  APG-66  radar  development  program,  relative 
to  LCC,  was  the  TWT.  Problems  identified  on  earlier 
radar  programs  led  to  this  concern.  As  a  result, 
separate  tests  were  required  for  the  TWT.  (Ref:  F-16 
Case  Study  p.  IIA-16.) 

c.  F/A-18  -  MCAIR  purchase  orders  to  Hughes  contained  the 
life-cycle  cost  structure  and  design-to-cost  structure 
so  that  both  organizations  participated  in  the  analysis. 
Early  in  the  program,  MCAIR  set-up  an  "off-line"  team  of 
people  to  monitor  these  analyses  and  to  assure  proper 
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FIGURE  C-6.  BIT  Mechanization  Trade  Study 
(Ref.  F-15  Case  Study,  p.  IIC-7) 


,VU 


attention.  Proposed  changes  to  the  radar  receive  LCC 
analyses  via  the  line  F/A-18  organization  today. 


d.  FIREFINDER  -  Desig n-to-uni t-production-cost  was  one  of 
the  principal  design  objectives  of  the  AN/TPQ-36  radar. 

A  $700,000  award  fee  was  established  for  this  purpose. 
The  objective  was  achieved  by  rigorous  examination  of 
all  production  costs  while  simultaneously  ensuring  that 
the  performance  and  R&M  requirements  would  be  met. 

These  examinations  were  done  not  only  at  the  hardware 
design  and  implementation  level,  but  at  earliest  stages 
of  conceptual  design  (Ref.  FIREFINDER  Case  Study,  p.  96) 


LDNS  -  A  life-cycle  cost  analysis  was  conducted.  The 
life-cycle  cost  was  defined  as  R&D  cost,  plus  acquisi¬ 
tion  costs,  plus  ten  years  of  operating  and  maintenance 
costs.  Design  and  maintenance  trade-offs  (RIW  versus 
Organic)  were  considered  in  determining  the  minimum 
cost  of  this  equipment  over  its  10-year  life  span. 

Usage  of  the  equipment  in  all  phases  of  the  life  pro¬ 
file  was  considered.  Cost-quantity  relationships  and 
risk-uncertainty  criteria  were  developed  as  part  of 
the  LCC  analyses  (Ref.  LDNS  Case  Study,  p.  24-25). 
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B.  MANAGEMENT 


The  contractor  organizations  involved  in  the  case  study 
systems  varied  considerably.  Most  followed  a  matrix  management 
approach  wit  .  the  contractor  R&M  personnel  selected  from  a 
functional  organization. 

Prime  contractor  R&M  management  in  the  F-15  program  was 
headed  up  by  an  engineering  chief  who  in  turn  reported  to  the 
Director  of  Engineering.  As  the  program  progressed  through 
the  various  phases,  the  backgrounds  of  the  respective  R&M  chiefs 
were  complementary  to  the  respective  phases  of  the  program. 

In  the  1969  time  frame  (early  in  the  program)  the  R&M  chief's 
background  was  largely  operations  and  systems  analysis  which 
complemented  early  program  activities  of  trade  studies,  predic¬ 
tions,  plans  and  their  impacts  on  life-cycle  costs.  During 
the  1973  time  period,  an  R&M  chief  with  an  avionics  background 
complemented  the  integration  of  radar  design  and  testing  efforts 
including  MIL-STD-781B  activities.  In  the  1976  and  later 
periods,  a  chief  with  a  background  in  laboratory  and  flight 
testing  complemented  test  activity  integration  including 
production  reliability  tests  with  the  resultant  product  im¬ 
provements  in  radar  R&M  (Ref.  F-15  Case  Study,  p.  IIB-10). 
Although  this  type  of  evolutionary  R&M  management  history  was 
not  detailed  in  the  other  case  study  reports,  it  is  presented 
here  based  only  on  F-15  data  as  it  does  serve  to  point  out 
that  as  a  program  passes  through  various  stages,  areas  of 
activity  and  focus  do  change  and  should  be  considered  in  selec¬ 
tion  of  managers  for  the  various  functions  in  the  various 
phases. 
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1 .  Planning,  Control  and  Emphasis 

In  looking  across  successful  R&M  programs,  one  of  the  important 
questions  was  the  role  of  planning  and  control  in  the  R&M  process 
and  the  emphasis  placed  on  R&M.  From  these  reviews,  it  is  evident 
that  the  emphasis  placed  on  R&M  by  the  government  in  the  entire 
program  process  from  conception,  RFP ,  source  selection,  FSD  and 
production  is  probably  the  single  most  important  driver  to 
achieving  reliable  and  maintainable  weapon  systems. 

Ways  of  providing  emphasis  are  discussed  both  in  this 
section  and  under  contracting  (para.  a).  Once  real  emphasis  on 
R&M  was  established  by  the  government,  the  contractor  top  manage¬ 
ment  reflected  this  emphasis,  and  a  better  balance  was  provided 
between  R&M,  cost,  schedule  and  technical  performance.  Once 
emphasis  was  established,  appropriate  planning  and  control 
tools  were  activated  and  reasonable  results  were  achieved. 

This  section  provides  some  insight  into  the  detailed 
analysis  across  several  successful  programs  and  approaches 
taken  to  emphasize  R&M  and  plan  and  control  the  associated 
program  process.  High-level  emphasis  on  R&M  and  a  closed  loop 
data  feedback  process  were  vital  to  the  overall  R&M  success. 

a.  Planning  &  Control 

Program  planning  and  control  can  influence  the  outcome  of 
any  event.  Within  the  F-15  radar  program,  technical,  cost  and 
schedule  requirements  and  controls  were  established  between  the 
prime  contractor  and  the  government  and,  in  turn,  passed  on  to 
suppliers  through  contract  documentation. 

Reliability  and  Maintainability  program  plans  were  required 
and  control  was  exercised  through  formal  review,  data  approval, 
and  considerable  personal  contact  between  management  and 
engineering  personnel.  R&M  approval  (signature)  was  required 
for  release  of  procurement  specifications,  installations  and 
assembly  drawings,  development  test  procedures  and  reports  as 
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well  as  suppliers'  data  requirements  (Ref.  F-15  Case  Study, 
pp.  IIB-3  and  IIB-9. ) 

Within  the  F-16  radar  program,  similar  controls  were  enacted 
requiring  R&M  drawing  signoff  as  well  as  participation  in  the 
configuration  control  board,  failure  review  board  and  special 
corrective  action  teams.  Of  particular  note  was  the  special 
planning  and  control  directed  toward  the  potentially  high-risk 
TWT  development  effort.  (Ref.  F-16  Case  Study,  pp.  IIB-14  and 
IIB-18.)  A  key  factor  demonstrated  here  is  the  ability  to  iden¬ 
tify  and  quantify  potential  high-risk  items  and  subsequently 
plan  and  control  the  effort  to  substantially  reduce  risk. 

For  planning  and  control,  the  F/A-18  radar  program  used  many 
concepts  similar  to  those  used  on  the  F-15  and  F-16  radar  programs. 
To  provide  detailed  visibility  to  the  responsible  managers, 
subsystem  status  charts  were  maintained  which  tracked  performance, 
reliability,  maintainability  and  cost  factors.  (Ref.  F/A-18  Case 
Study,  pp.  97  and  98.) 

To  provide  additional  R&M  visibility  and  control,  the  F/A-18 
radar  program  conducted  R&M  program  review  meetings  which  were 
attended  by  high-level  government  personnel.  (Ref.  F/A-18  Case 
Study,  p.  94.) 

Although  the  initial  TPQ-37  program  had  difficulties  with 
the  implementation  of  a  reliability  improvement  program  in  1977, 
planning  and  control  factors  were  initiated.  The  program  included 
elements  as  seen  in  the  other  case  studies  such  as  program  reviews 
and  high-level  failure  review  board  activity.  (Ref.  FIREFINDER 
Case  Study,  p.  81.) 

Across  the  programs  analyzed,  the  quality  assurance  provisions 
were  similar.  Although  mechanization  varied  from  case  to  case, 
the  objectives  were  essentially  the  same. 

An  important  thread  which  stretched  across  the  successful 
programs  was  the  provision  for  timely  data  feedback  which  allowed 
early  detection  of  problems  and  appropriate  adjustments  to  the 
planning  and  control  process  to  effect  solutions. 
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b.  Emphasis 

Management  emphasis  was  placed  on 
avenues  across  the  programs  analyzed, 
and  other  related  emphasis  factors  are 
paragraphs. 


R&M  through  a  number  of 
Reporting  relationships 
discussed  in  the  following 


( 1)  Reporting  Relationships 

One  of  the  methods  of  attempting  to  convey  R&M  emphasis  in 
the  past  has  been  the  elevation  of  the  R&M  function  within  the 
overall  organizational  hierarchy.  Varying  degrees  of  this  are 
observable  across  the  case  studies  conducted. 

For  the  F-15  program,  the  prime  contractor  organization  had 
the  R&M  chief  reporting  to  the  Director  of  Engineering.  Their 
radar  subcontractor  had  the  R&M  function  reporting  to  the  Radar 
Program  Manager  (Ref.  F-15  Case  Study,  pp.  III-B-10  and  II-B-12). 

For  the  F-16  program,  which  was  an  RIW  contract,  there  was 
a  somewhat  different  approach  in  that  both  the  prime  and  sub¬ 
contractor  each  had  RIW  program  managers  which  reported  to  the 
vice-presidential  level  with  matrixed  interfaces  to  the  Radar 
Program  Manager  who  also  reported  to  the  vice-presidential  level. 
In  addition,  the  reliability  function  at  the  subcontractor 
reported  through  the  Quality  and  Reliability  Assurance 
Organization  as  opposed  to  the  Director  of  Engineering  as  was 
the  case  on  the  F-15.  (Ref.  F-16  Case  Study,  pp.  II-B-5  and 
II-B-7. ) 

Although  organizational  structuring  was  not  provided  in 
the  F/A-18  and  FIREFINDER  case  study  documents,  the  AEGIS 
program  does  reflect  another  approach  to  organizational  hierarchy 
On  the  AEGIS  program,  the  reliability  function  reported  through 
the  System  Engineering  function  to  the  RCA  AEGIS  program  manage¬ 
ment  office. 


Perhaps  the  whole  question  relative  to  the  optimum  place¬ 
ment  of  the  Reliability  and  Maintainability  functions  was  best 
related  by  a  high  corporate  officer  of  a  major  California 
corporation  in  discussions  at  the  corporation's  first  annual  R&M 
symposium  held  toward  the  end  of  1982.  At  this  symposium,  all  of 
the  R&M  directors  of  all  the  subcompanies  within  the  corporation 
convened  near  Los  Angeles  for  a  two-day  meeting.  In  the  meetings, 
each  company  addressed  how  the  R&M  function  reported  within  their 
company.  Not  unexpectedly,  the  results  covered  the  spectrum  of 
possibilities — including  the  Engineering,  Quality  Assurance, 

Product  Support,  Product  Safety,  Quality  Engineering,  Product 
Assurance,  Logistics  Integration,  Systems  Effectiveness,  Commercial 
Support,  Advanced  Systems  and  Technical  Services  Organizations, 
just  to  name  a  few. 

When  the  corporate  executive  was  queried  as  to  whether  there 
was  any  apparent  correlation  with  R&M  performance  of  the  various 
companies  and  their  unique  reporting  structure,  he  replied  that 
if  so,  it  was  not  readily  relatable.  In  follow-up  conversations, 
he  related  that  he  had  come  to  understand  that  the  person  in  the 
job  was  the  important  factor,  not  how  the  job  reported.  In 
summary,  he  stated  that  a  "doer"  who  is  interested  and  supportive 
of  R&M  can  make  things  happen  in  almost  any  reporting  structure 
within  the  formal  and  informal  organizational  hierarchy. 

( 2 )  Team  Concept 

A  major  thread  which  surfaces  looking  across  successful 
programs  is  establishment  of  a  non-adversar ial  relationship  with 
heavy  emphasis  on  a  team  approach  to  problem  investigation  and 
resolution.  This  approach  was  strongly  emphasized  on  the  F-16 
radar  from  the  earliest  establishment  of  requirements  throughout 
the  entire  program.  Confrontational  negotiations  were  avoided 
and  the  notion  of  team  effort  was  strongly  supported  at  all 
levels.  This  concept  was  even  extended  to  suppliers  where 
the  subcontractor  provided  assistance  to  suppliers  to  ensure 
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their  products  were  up  to  the  required  quality  levels.  Moti¬ 
vational  meetings  were  held  with  supplier  and  employees,  awards 
presented  and  F-16  films  shown.  (Ref.  F-16  Case  Study,  p.  II 
B-16 .  ) 

For  the  F/A-18  radar  program  steps  to  promote  the  team  concept 
included  collocation  of  personnel  within  the  same  area.  For 
example  at  the  prime  contractor's  facility  the  R&M  engineers  were 
collocated  with  the  design  engineers.  (Ref.  F/A-18  Case  Study, 
p.  96.) 


2 .  Monitoring  and  Control  of  Subcontractors 

The  whole  can  only  be  as  good  as  its  weakest  part.  With 
the  trend  in  the  recent  decade  toward  increased  levels  of  sub¬ 
contracting  from  the  prime  contractor  level,  the  R&M  role  of 
monitoring  and  control  of  subcontractors  and  suppliers  becomes 
increasingly  important.  Within  each  of  the  successful  programs 
reviewed,  particular  attention  was  paid  to  the  performance  of 
subcontractors  and  suppliers. 

Typical  of  the  things  done  are  the  following  items  from 
the  FIREFINDER  Program.  R&M  requirements  were  allocated  to  the 
subcontractor  via  a  procurement  specification.  The  prime  con¬ 
tractor  conducted  design  reviews  and  performed  thermal  studies 
and  R&M  analyses  prior  to  qualification  testing.  Stress  screens 
were  developed.  A  failure  feedback  and  corrective  action  system 
was  imposed  on  the  complex/critical  items  suppliers,  and  pro¬ 
duction  control  testing  was  monitored  to  ensure  failures  attri¬ 
buted  to  subcontractor  items  were  reviewed  by  Failure  Review  Boards 
(Ref.  FIREFINDER  Case  Study,  p.  84.) 

In  the  case  of  the  F-16  radar  program,  identification  of 
the  TWT  as  a  potentially  high  risk  item  and  life-cycle  cost 
driver  generated  the  requirement  for  special  management  attention. 
As  a  result  the  government  and  prime  contractor  jointly  devised 
a  reliability  test  for  the  TWT  and  included  it  in  the  RFP  to 


the  radar  subcontractor.  The  radar  subcontractor  in  turn  imposed 
a  reliability  growth  test  requirement  on  its  subcontractor  for 
the  TWT.  As  a  result  of  this  special  attention  the  high-risk 
item  was  successfully  managed  and  a  field  performance  over 
four  times  higher  than  the  predicted  TWT  MTBF  was  achieved. 

(Ref.  F-16  Case  Study,  pp.  II  B-18  and  II  B-19.) 


3.  Engineering  Change  System 

In  order  to  produce  reliable  and  maintainable  weapon 
systems,  a  formal  controlled  process  must  exist  to  provide  for 
detecting  problems,  determining  solutions,  generating  changes 
and  then  implementing  changes  into  the  respective  hardware  and/or 
software. 

A  common  thread  across  the  successful  programs  reviewed 
both  by  case  study  and  report  research  was  flexibility  in  the 
change  process  system  and/or  accelerated  approval  of  engineer¬ 
ing  changes.  Within  the  F-16  radar  program,  this  was  accomplished 
by  processing  R&M  type  changes  both  as  Class  I  and  Class  II  re¬ 
sulting  in  excess  of  5700  changes.  (Ref.  F-16  Case  Study, 
p.  ID-11.) 

In  an  attempt  to  quantify  ECP  proposal  flow,  data  were 
collected  for  several  programs  (see  Fig.  C-7 ) . 

ECP  PROPOSAL  FLOW 

Change  Initiation  Through  Contractual  Approval 


Program 

Average  Flow 

Calendar  Days 

E-3A 

344 

E-4 

210 

ALCM 

164 

Roland 

178 

LDNS 

5-30 

FIGURE  C-7.  ECP  Approval  Cycle 


89/11-1 


V  V. 


C.  DESIGN 

This  section  describes  the  processes  used  on  the  case  study's 
systems  to  influence  the  design  and  to  effect  the  R&M-related 
design  features. 

1 .  Development  of  Design  Requirements 

Four  of  the  six  case  studies  that  discussed  the  development 
of  detailed  design  requirements  started  with  the  translation  and 
apportionment  of  platform  R&M  requirements  to  avionics  requirements 
which  were  then  passed  to  the  subcontractors  in  very  detailed 
specifications.  On  the  F-15,  F-16  and  F/A-18  radars  programs,  the 
prime  contractor  was  allowed  to  reapportion  requirements,  or 
else  required  design  optimization  via  design  alternative  studies 
performed  by  the  subcontractor.  The  FIREFINDER  program's  two 
radars  were  prime  contracts  to  the  Army,  who  provided  the  per¬ 
formance  and  R&M  requirements  directly  to  the  contractor.  LDNS 
R&M  design  requirements  were  developed  from  user  needs  and  trans¬ 
mitted  to  the  contractor  in  the  development  specification.  The 
contractor  took  a  top-down  approach  where  a  MTBF  number  was 
allocated  to  each  subassembly. 

Specific  design  guides  were  prepared  for  the  designers  in 
the  areas  of: 

e  Parts  and  material  selection 
e  Derating 
e  Design  practices 

e  Packaging  of  electronic  equipment 

e  Partitioning  and  test  pointing  for  compatibility  with 
automatic  test  equipment. 

The  source  of  these  guides  ranges  from  guidance  documents  pro¬ 
vided  by  the  prime  contractor,  to  allowable  parts  lists,  to  military 
specifications,  standards  and  design  guides.  In  all  cases,  it  was 
indicated  that  these  served  as  a  starting  point  only,  and  that 
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The  range  for  the  period  from  change  initiation  through  contractual 
approval  was  5  to  344  average  flow  calendar  days.  Recogniz¬ 
ing  that  activity  cannot  really  commence  until  change  is 
approved,  an  inhibitor  to  improving  R&M  is  built-in.  Where 
there  are  high  production  rates,  this  can  lead  to  costly  retro¬ 
fits  and  fielding  of  hardware  with  known  R&M  problems.  This 
then  contributes  to  the  overall  field  data  and  tends  to  lower 
the  overall  reported  R&M  levels. 

In  the  case  of  the  LDNS,  extraordinary  procedures  were 
used  to  minimize  the  time  required  for  ECP  approval.  Frequently, 
Army  approval  was  obtained  within  5  days.  In  all  cases,  the 
ECP  request  was  acted  upon  within  30  days. 
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considerable  concurrent  analysis,  design  alternative  studies,  design 
feedback,  and  design  adjustments  were  made.  Strict  interpretation 
of  many  of  these  guides  allowed  exceptions  to  surface  and  be 
effectively  managed. 

Design  alternatives  were  predominant  in  choice  of  components, 
thermal  and  environmental  considerations  and  addressed  primarily 
the  reliability  requirements.  Designing  of  the  Built-In-Test 
(BIT) ,  and  the  off-line  testability  requirements  benefited  from 
the  formal  and  several  automated  analytical  techniques  presented. 

Unlike  specific  direction  in  parts  choice  and  derating, 
testability  related  design  features  evolve  from  general  design 
guides,  testing  of  resulting  ideas  on  paper  and  adjusting  and  fine 
tuning  the  ideas,  until  there  is  reasonable  confidence  that  a 
requirement  can  be  met.  At  risk  is  costly  redesign. 

The  process  begins  with  system  partitioning  into  lower-level 
assemblies  and  analyzing  the  ability  of  fault  detecting  and 
isolating  to  the  functional  modes  created  by  the  partitions,  with 
either  test  points  or  BIT  facilities.  Partitioning  was  invariably 
in  competition  with  packaging  density,  availability  of  connector 
pins,  size,  thermal  and  performance  requirements.  The  F-15,  F/A-18 
and  FIREFINDER  case  studies  describe  how  computer  techniques  were 
used  to  provide  rapid  analyses  and  feedback  to  the  designer  and 
to  guide  him  into  developing  the  appropriate  design. 

2.  Design  Alternative  Studies 

Several  deliberate  R&M-related  trade  studies  were  conducted 
for  the  F-15,  F-16  and  F/A-18  and  FIREFINDER  radars  and  the  LDNS . 

The  studies  examined  alternatives  with  which  to  best  meet,  rather 
than  improve  upon,  the  specified  reliability,  maintainability, 

BIT  and  testability  requirements.  Major  issues  considered  in 
the  studies  were  risk  in  meeting  specifications,  compatibility 
with  performance  requirements,  ease  and  practicality  of  implemen¬ 
tation  as  well  as  minimizing  design  impact.  Figure  C-8  is  a 
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summary  of  the  more  significant,  deliberate  design  alternatives 
that  were  reported. 


SYSTEM 

QUANTITY 

CASE  STUDY  PAGE 

F-15 

5 

IIC-7 

F-16 

7 

IIC-9,  11 

F-18 

6 

111 

FIREFINDER 

9 

56/2-8  thru  -10 

FIGURE  C-8.  Deliberate  R&M  Design  Alternatives 

Whereas  the  deliberately  implemented  studies  may  have  provided 
improvements  beyond  specified  values,  significant  improvements 
were  realized  as  a  by-product  from  alternative  design  studies 
conducted  for  performance  enhancement,  weight  reduction,  cost 
reduction,  thermal  considerations,  engineering  change  proposals 
or  similar  reasons.  These  are  summarized  in  Fig.  C-9. 


SYSTEM 

F-15 

F-16 

F-18 

FIREFINDER 


QUANTITY 

11 

5 

3 

17 


CASE  STUDY  PAGE 
IIC-11  thru  21 
IIC- 13 

112  thru  114 
56/2-8  thru  10 


FIGURE  C-9.  Design  Alternatives  Resulting  in 

R&M  Benefits 
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The  summaries  indicate  that  the  studies  always  considered  R&M 
impact  and,  in  turn,  life-cycle  cost  impact,  and  usually  resulted 
in  a  substantial  improvement.  Examination  of  the  design  details 
evaluated  in  the  studies  indicates  that  design  improvements,  by 
applying  the  latest  proven  state-of-the-art  components,  packaging 
and  cooling,  as  well  as  software  techniques,  caused  improvements 
in  R&M.  Figure  C-10  graphically  depicts  the  reliability  improve¬ 
ments  that  were  realizable  from  technology  growth  of  electronic 
components . 

The  data  indicate  that  trade  studies  can  have  a  significant 
impact  in  establishing  the  R&M  design  baseline  and  an  even  greater 
impact  on  R&M  growth  as  part  of  a  design  maturation  and  improvement 
program.  In  planning  such  growth,  an  aggressively  managed  trade 
study  program  directed  at  finding  a  proper  balance  between  the 
design  issues  and  the  potential  gains  in  R&M  is  essential  to 
effect  R&M  benefits  from  design  improvements. 


89/14-4 


CO  CO 

</)  </> 

®  t-  CO  CO 

</} 

OC  h- 

ZQ-  S  K  CO 

h 

Zd. 

0:0.  uj 

d- 

foe  ec 

ZT  C 

CslKN  = 

C 

CMC 

oc  tni — 

c 

LO 

•>  lo 

LT\ 

oc  , 

O* 

*— 1  N"\ 

««««!  *f  ■ft  .  iK.^srtvRk 

iaa^aaaaajgafe 

--L  r,'z~:  I'.l.'l  mf_~. 

BWOI^-  jfcaUc-uPTdCJ 

5SS^55£B^^Ss= 


.  _ _ _.,jBiiiiDBiiiimiiiiaiaa 

|  ■MHMtrioiuiuuiiHuMixiiHaiiiMoaa* 
■muonnuiiunipiraitiiiiaiii 

unir.uiii,  lumaBMaHHiumiu'u 
pfMr.uiiH 
bPMUMlUl 
I^UMIBO 

14  t 1  i 


Hxvr^MflBia 

bi  nr  is  mu  win 
iramasumuj^l 


*i:i. 


DCrmt*ralxii*tLCi«*fejr*ince«5****i^Efrw  cm 


■  ■ivMHnaMBmavTTinBDiiinnni 

■  ■iriiinBaimiiinininiM 

■raflMuniBia— mumhummm 


FIGURE  C-10.  Complexity  and  Parts  Technology  Drive  Reliability 


Design  Evaluation  Analysis 


All  case  studies  reported  the  use  of  design  evaluation 
analyses  as  part  of  the  design  process  and  design  alternative 
studies.  The  results  of  the  analyses  were  employed  in  component 
and  material  selection,  establishing  stress  levels,  and  in 
mechanical  and  electrical  design.  Fig.  C-ll  summarizes  the 
techniques  reported. 

The  case  studies  do  not  provide  details  of  the  techniques 
used,  from  which  it  may  be  assumed  that  military/industry  standard 
techniques  were  used  except  for  the  computerized  techniques 
(asterisks  on  the  figure). 
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4.  Parts  and  Material  Selection  and  Control 


A  formal  parts  and  material  control  process  was  followed  by 
the  F-15,  F-16,  F/A-18  and  FIREFINDER  programs  with  each  placing 
slightly  different  emphasis  on  what  was  controlled,  and  the 
degree  of  the  control  to  effect  the  specified  reliability  and 
standardization  requirements.  Figure  C-12  provides  an  overview 
of  that  control.  The  prime  contractor  chaired  the  parts  control 
board  for  all  three  subcontracted  radar  programs. 
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FIGURE  C-12.  Parts  Control  Overview 
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The  following  summarizes  the  emphasis  given  to  the  parts 
and  material  programs  as  stated  in  the  case  studies. 

a.  F-15  -  A  parts  program  for  all  contractor-furnished 
equipment  was  established  in  the  MCAIR  reliability 
program  plan  and  implemented  by  specific  requirements 
in  the  procurement  specification.  Control  was  exer¬ 
cised  by  MCAIR  review  and  approval  of  parts.  A  preferred 
parts  list  was  submitted  by  Hughes.  Program  instruc¬ 
tions  were  issued  which  mandated  parts  selection 

and  controls  procedures,  parts  electrical  and 
thermal  derating  criteria  and  a  weekly  update  of 
computerized  module  indentured  parts  lists  for 
review  and  control  of  parts  selection  (Ref.  F-15 
Case  Study,  pp.  IIC-68,72). 

The  process  led  to  standardization  in  module 
construction  with  four  basic  types;  standard  parts 
and  wiring  access  covers;  standardized  module  restraint, 
identification  and  keying,  and  a  standard  cooling 
system  using  an  integral  module  heat  exchanger. 

b.  F-16  -  General  Dynamics  imposed  aggressive  parts 
control  and  standardization  requirements  on  all  new 
designs  by  specific  requirements  in  the  equipment 
specifications  and  statements  of  work.  Full  govern¬ 
ment  support  was  provided  to  develop  new  military 
standards  for  multiple  use  parts.  The  military 
standards  available  for  designers  to  select  from 
were  reduced  to  those  with  established  reliability 
requirements. 

Commonality  was  forced  by  reducing  the  number  of 
standards  available  to  designers  and  subcontractors. 

The  F-16  Program  Parts  Selection  List  was  established 
after  a  comprehensive  review  of  military  specifica¬ 
tions.  The  number  of  standards  was  reduced  to  only  those 
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with  Established  Reliability  (ER)  requirements.  The  F-16 
Parts  Control  Board  maintained  constant  contact  with 
Rome  Air  Development  Center  (RADC)  for  state-of-the-art 
device  recommendations.  Commonality  was  achieved  by 
selecting  standards  based  on  RADC  recommendations. 

The  F-16  Parts  Control  Board  reviewed  the  microcir¬ 
cuit  industry  and  established  that  the  low  power 
Shottky  technology  would  be  the  leading  technology 
for  the  1980s.  The  54H  and  54L  technologies  were 
eliminated  from  the  PPSL.  The  dual-in-line  microcir¬ 
cuit  package  was  selected  over  the  flat  pack  for  the 
PPSL. 

Fastener  types  were  reduced  from  226  to  47  and 
the  fastener  recess  standardized.  Limiting  the  use 
of  high  failure  rate  electromechanical  parts  resulted 
in  use  of  only  six  potentiometers,  two  relays  and 
four  motors. 

c.  F/A-18  -  An  extensive  parts  control  program  was  estab¬ 
lished  by  MCAIR  and  approved  by  NAVAIR.  The  program 
established  parts  derating  requirements  that  were 
more  stringent  than  NASA  guidelines  which  resulted 

in  a  high  percentage  of  high  reliability  and  standard 
parts  in  the  radar  (Ref.  F/A-18  Case  Study,  pp.  45B/ 
7-21). 

d.  FIREFINDER  -  Much  of  the  success  of  FIREFINDER  relia¬ 
bility  design  effort  was  due  to  the  up-front  planning 
of  consistent  and  standardized  requirements  which  were 
clearly  disseminated  to  the  designer.  Allocations 
were  made  for  reliability  which  included  preferred 
parts  and  materials  and  detailed  standards  for  printed 
wiring  boards.  A  producibility/standardization  guide 
was  also  created  at  the  beginning  of  the  radar  design 
phase  containing  guidelines  for  parts  selection  and 
specified  allowable  stress  levels  for  all  major  com¬ 
ponent  classes  (Ref.  FIREFINDER  Case  Study,  p.  100). 
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Two  approaches  to  parts  control  were  employed. 

The  first  was  a  strict  array  parts  control  plus 
non-standard  parts  review  and  approval  by  DESC 
for  the  AN/TPQ-36  antenna.  This  resulted  in  a 
2.3  to  1  ratio  of  MIL  standard  to  non-standard  parts 
with  no  parts  upgrade  required.  The  approach  used  for 
the  AN/TPQ-37  antenna  employed  industrial  standard  and 
GSG  standard  parts  without  requiring  non-standard  parts 
approval.  This  resulted  in  a  1.84  to  1  ratio  of  MIL 
standard  to  non-standard  part  types,  but  required 
46  semi-conductor  upgradings  to  meet  temperature  range 
requirements;  47  components  required  upgrading  by  burn- 
in  prior  to  acceptance  tests  and  alternate  source 
development  for  63  parts.  The  AN/TPQ-36/37  common 
shelter  was  influenced  by  both  AN/TPQ-36/37  and  as  a 
result,  achieved  a  ratio  of  1.98  to  1  (Ref.  FIREFINDER 
Case  Study,  pp.  94,  95). 

LIGHTWEIGHT  DOPPLER  NAVIGATION  SYSTEM  -  The  MIL-P- 
11268  specification  was  a  contract  requirement.  The 
parts  control  and  selection  was  managed  through  a  govern¬ 
ment/contractor  parts  control  board.  The  use  of  standard 
parts  was  stressed.  Deviation  from  part  selection  and 
material  selection  control  was  granted  by  government 
(project  engineer  or  project  manager).  The  contractor  was 
required  to  show  cost  advantage  or  reliability  improve¬ 
ment.  Project  personnel  control  of  the  board  expedited 
the  selection  process.  Configuration  control  was  not 
enforced  until  the  completion  of  testing  to  alleviate 
cumbersome  procedures  (Ref.  LDNS  Case  Study,  p.  65/1-17). 
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5.  Derating  Criteria 

The  component  derating  criteria  for  the  four  case  studies 
were  established  by  a  reliability  policy/plan  as  required  by  the 
individual  contracts.  Stress  levels  were  based  on  standard  levels 
developed  either  by  the  prime  contractor  or  manufacturer  with  no 
details  provided  for  tailoring  to  a  specific  application,  as 
follows: 

a.  F-15  -  stress  levels  were  contractually  imposed  by  MCAIR's 
reliability  program.  Changes  required  MCAIR  approval  (Ref.  F-15 
Case  Study,  p.  II  C-78). 

b.  F-16  -  stress  levels  set  by  the  subcontractor  with  General 
Dynamic's  approval.  Levels  were  tailored  to  the  environment  and 
intended  use  and  verified  by  stress  analysis. 

c.  F/A-18  -  standard  NASA  stress  levels  were  tailored  to  the 
application  based  on  trade-offs  made  during  the  proposal  phase  for 
the  radar.  The  levels  were  included  in  the  procurement  specification. 
Verified  by  stress  analysis. 

d.  FIREFINDER  -  the  derating  requirements  were  set  by  the  Army 
with  detailed  levels  established  by  the  manufacturer's  producibility/ 
standardization  requirements  as  part  of  his  reliability  program. 

These  were  provided  to  the  designer  in  the  form  of  derating  and 
application  guidelines.  Less  than  0.1  percent  of  overstress  was 
reported  from  testing  and  operational  use  and  these  were  cleared 
with  relatively  little  impact  on  the  program  (Ref.  FIREFINDER 

Case  Study,  p.  106). 
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a.  Derating  Parameters 

The  electrical  parts  derating  levels  for  circuit  components 
are  listed  in  Fig.  C-13  for  the  AN/APG-63,  AN/APG-66  and  AN/APG-65 
radars.  Details  for  the  FIREFINDER  antenna/transceiver  and  common 
shelter  were  not  available.  In  addition,  the  AN/APG-63  lists  the 
following: 

•  Relays:  10%  with  a  lamp  load 

20%  with  an  inductive  load 
40%  with  a  resistive  load 

•  Switches  50%  of  rated  current 

•  Motors  50%  of  rated  load 

•  Gears  60%  of  rated  load 

•  Bearings  60%  of  rated  load 

(Ref.  F-15  Case  Study  pp.  II  C-83,  89,  90). 
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b.  Exceptions  to  Derating  Criteria 

Only  the  AN/APG-65  (F/A-18)  case  study  reported  derating 
exceptions,  listing  those  applications  with  stresses  in  excess  of 
103  percent  of  the  derate  value.  Figure  C-14  lists  the  parts 
involved  together  with  the  reason  alternate  parts  could  be 
used.  Seventy-five  of  the  94  parts  required  exceptions  due  to 
overstressing  and  19  due  to  overtemperature.  Many  of  the  over¬ 
temperature  problems  were  solved  by  thermal  packaging  redesign. 

The  94  parts  represent  less  than  0.7  percent  of  the  total 
parts  count  of  13,500. 
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(Ref.  F/A-18  Case  Study  pp.  131-136). 

FIGURE  C-14.  AN/APG-65  Derating  Exceptions 
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Thermal  and  Packaaina  Criteria 


The  thermal  and  packaging  criteria  were  based  on  the 
allowable  parts  temperature,  environmental  requirements,  cool¬ 
ing  air  availability  and  restrictions  on  construction.  This  is 
illustrated  graphically  in  Figure  C-15,  the  thermal  design  con¬ 
sideration  for  a  module  taken  from  the  F-15  case  study.  Restric¬ 
tions  to  the  design  freedom  for  thermal  considerations  included 
size,  weight  and  shape  restrictions;  vibration  considerations 
material  priorities  for  structural  considerations;  parts  types 
and  their  mounting  requirements;  and  manufacturing  techniques. 

Trade-offs  were  made  to  address  all  of  these  requirements, 
from  which  the  final  designs  evolved.  The  major  areas  of  thermal 
design  considerations  presented  in  the  case  studies  were: 

a.  F-15 

•  Integral  module  heat  exchanger  selected  for  dramatic 
thermal  performance  improvement. 

•  Flatpacks  and  surface  mounted  discretes  selected 
for  their  lower  thermal  impedance,  easier  removal 
and  higher  packaging  density. 

•  Bonded  crushed  honeycomb  design  selected  for  both 
light  weight  and  a  reduced  transmissibility  of  3  to 
5  vs.  approximately  20  for  conventional  designs. 

•  Cooling  air  is  introduced  at  unit  rear  panel 
and  exhausted  through  bottom  cover. 

•  Central  (or  side)  unit  air  plenum  distributes  air 
to  all  modules  in  parallel. 

•  Air  is  metered  by  orifices  in  module  inlet  manifold. 
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•  Inlet  and  exhaust  manifolds  provide  uniform  air 
flow  through  manifold  by  establishing  equal  length 
flow  path  (Ref.  F-15  Case  Study,  pp.  48-52). 

b.  F-16 

•  High  power  components  were  placed  nearest  the  edge 
or  coolest  part  of  the  subassembly. 

•  Copper  heat  sinks  were  used  rather  than  aluminum 
in  several  critical  locations. 


•  Circuit  boards  were  arranged  for  efficient  cooling. 

•  Cooling  air  was  apportioned  between  LRUs. 


c.  F/A-18 


Refinement  of  the  F-15  heat  exchanger  consisting  of 
a  lightweight  aluminum  heat  exchanger  sandwiched  be¬ 
tween  two  multilayer  printed  wiring  boards  affords  15° 
to  20°C  temperature  reduction  of  components  (Ref.  F/A- 
18  Case  Study,  p.  120). 


d.  FIREFINDER 

Blowers  and  air  ducts  through  cold  plates  were  used 
to  cope  with  desert  environment.  The  higher  power 
AN/TPQ-37  transmitter  required  liquid  cooling  (Ref. 
FIREFINDER  Case  Study,  p.  110). 
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Computer-aided  design  (CAD)  is  an  analytical  tool  which 
can  automatically  evaluate  and  optimize  a  design  attribute. 

Until  recently,  it  has  been  used  primarily  for  system/equip¬ 
ment  structural  and  performance-related  design  requirements. 
Though  the  case  studies  do  not  address  the  topic  directly,  evi¬ 
dence  of  the  application  of  some  level  of  CAD  for  R&M  is  found 
in  the  description  of  analytical  processes  for  the  F-15,  F-16, 
F/A-18  and  LDNS,  as  follows: 

a.  The  F-15  program  reports  the  use  of  computerized 
thermal  analysis  and  design,  BIT  analyses  and  auto¬ 
matic  test  generation. 

b.  The  F-16  program  reports  a  computerized  optimum 
repair  level  analysis  as  a  trade-off  tool  and  a 
computerized  analyses  thermal  and  computerized  produc¬ 
tion  in  the  radar's  design. 

c.  The  F/A-18  program  describes  the  use  of  a  computerized 
sneak  circuit  analysis  and  thermal  analysis  for 
circuit  design. 

d.  The  LDNS  reports  the  use  of  computerized  worst-case 
analysis. 

The  FIREFINDER  and  AEGIS  programs  do  not  describe  CAD 
techniques. 

Computer-aided  design  for  R&M  has  rapidly  expanded  to 
include  interactive  R&M  predictions,  the  evaluation  of 
electrical,  thermal  and  mechanical  stresses,  sneak  paths, 
test  point  planning,  BIT  strategy  and  software  design,  choke 
points,  testability  and  other  issues  that  can  aid  or  pro¬ 
vide  automatic  decisionmaking  for  forward-going  design  or 
the  evaluation  of  design  alternatives.  Computer-Aided  Design 
(CAD)  examples  from  the  case  studies  dealt  with  specific  areas. 
CAD  had  not  yet  evolved  to  the  point  that  enabled  linking 
many  of  these  areas  into  an  automated  interactive  design  process 
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The  details  presented  in  the  case  study  of  the  few  appli¬ 
cations  of  R&M-related  CAD  that  were  discussed  indicate  a  high 
potential  payoff  in  the  application  of  contemporary  and  new 
CAD  technology  in  the  design  process,  particularly  in  the  area 
of  BIT,  testability  and  diagnostics  preparation. 

8.  Testability  Analysis.  All  six  case  studies  describe 
the  process  of  developing  the  design  in  the  area  of  testability, 
with  varying  degree  of  detail.  The  analytical  processes 
described  are  included  in  an  overall  iterative  design  require¬ 
ments/evaluation/analyses  process  forming  a  part  of  formal 
design  control  addressing  R&M. 

All  case  studies  used  formal  program  plans  and  formal 
prime  contractor/government  controls  which  specified  the  process 
and  all  indicate  that  the  specified  requirements  were  translated 
to  the  designer.  Though  no  timetable  is  given,  it  can  be 
assumed  from  the  availability  of  analytical  results  at  design 
reviews  that  design  direction  took  place  at  the  very  beginning 
of  the  programs. 

The  results  of  the  process  shown  in  Fig.  C-16  are  indica¬ 
tions  that  the  very  stringent  specified  requirements  were  demon¬ 
strated  to  have  been  met,  or  slightly  bettered.  The  F/A-18  case 
study  provided  a  brief  discussion  of  the  potential  of  the  BIT  to 
fault  isolate  to  subassemblies  (SRAs),  though  not  required,  does 
exist. 
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•  Mathematical  organizational  level 

evaluation  for  fault  isolation 
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Field/Depot  Testing: 

•  Computerized  digi-  Fault  detection 

tal  simulation  89.7%  to  99.9% 

and  test  and  isolation  to  single 

sequential  auto-  component  with  guided 

matic  test  gen-  probe 

erator 

(Ref.  F-15  Case  Study,  pp.  102-110). 


F-16 ,  AN/APG-66  BIT:  effectiveness 

analysis 

•  Formal  design 
reviews 

•  Checklists 

Field/Depot  Testing: 

•  Test  tolerance 
analysis 

•  Checklists 

F/ A- 18, AN/ APG- 6 5  BIT:  effectiveness 
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FIGURE  C-16.  Testability  Analysis  Summary 
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compatible 
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FIGURE  C-16.  cont'd 


9 .  Testability  Verification  and  Testing 

The  verification  and  testing  of  the  testability  attributes 
in  the  course  of  the  design  process  (as  distinguished  from  formal 
demonstration  and  field  tests),  was  accomplished  with  the  analy¬ 
tical  tools  described  in  subparagraph  8  above. 

The  F-16  case  study  reports  a  pre-demonstration  design 
evaluation  in  addition  to  the  analyses  by  inserting  1462  faults 
to  test  fault  detection  capabilities.  This  is  ten  to  twenty 
times  the  normally  demonstrated  quantity. 

The  F/A-18  case  study  describes  a  formal  analytical  verifi¬ 
cation  program  imposed  by  the  prime  contractor  and  the  FIREFINDER 
reports  a  deliberate  manual  testing  of  every  test  point  to  verify 
that  injected  faults  could  be  detected  by  BIT. 


D.  MANUFACTURING 


In  the  area  labeled  as  production,  the  case  studies 
examined  two  specific  areas  of  activity:  environmental  stress 
screening  and  the  failure  reporting,  analysis  and  corrective 
action  system.  Both  of  these  activities  span  the  development  as 
well  as  the  production  phase,  but  their  primary  focus  and  impact 
is  on  the  reliability  and  maintainability  of  manufactured  items, 
as  opposed  to  the  R&M  of  the  design  of  the  equipment.  In  examining 
these  areas,  the  case  studies  did  review  the  two  subjects  during 
the  development  phase. 

1 .  Environmental  Stress  Screening  of  Parts  and  Equipment 

Environmental  stress  screening  (ESS)  has  been  defined  as: 

"The  process  or  method  whereby  a  group  of  like  items  are  subjected 
to  physical  stress  to  identify  and  eliminate  latent  part  and 
manufacturing  defects  prior  to  field  deployment"  (Ref:  RADC  TR- 
82-87).  ESS  is  unlike  the  tests  that  are  normally  associated 
with  reliability  development  since  it  is  designed  to  stimulate 
the  precipitation  of  defects,  not  to  simulate  the  operational 
environment.  The  specific  screening  methods  are  tailored  to  the 
specific  part  or  workmanship  defects  that  are  expected  or  predicted. 

a.  Levels  of  ESS 

ESS  is  applied  at  various  levels  of  assembly;  piece  part, 
module,  unit,  and  system  level,  for  the  F-15,  F-16,  F/A-18  and 
FIREFINDER  radars.  Each  radar  contains  "hi-rel"  parts,  including 
MIL-M38510  microcircuits  which  are  extensively  screened  to  MIL- 
STD883  by  the  part  manufacturer.  In  addition,  each  radar  manufac¬ 
turer  conducts  on-receipt  testing  of  selected  components.  This 
includes  part  screening  comprised  of  100  percent  test-at-tempera- 
ture  (including  functional)  for  microcircuits,  PIND  (Particle  Impact 
Noise  Detection)  testing  for  large  cavity  devices,  (transistors 
and  diodes),  and  special  tests  on  selected  devices.  Screening  at 
the  module  (SRA/SRU),  units  (WRA/LRU )  and  set  (system)  are  con¬ 
ducted  differently  among  th  >  jur  radars.  Figures  C-17  through 
C-19  summarize  ESS. 
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FIGURE  C-17.  Stress  Screening  Use 
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The  significant  differences  are: 

e  At  the  module  (SRA/SRU)  level,  the  F-16  modules  were 
subjected  to  temperature  cycles  from  -40°C  to  +71°C  with 
two  minutes  of  random  vibration  for  every  twenty 
minutes  of  operation  and  then  it  was  verified  that  the 
module  works  at  low  (-40°C)  and  high  (+71°C) 
temperatures.  The  F-15  and  F/A-18  radar  modules  were 
subjected  to  temperature  cycles  at  greater  extremes 
(-60°  to  +95°C)  without  vibration  and  function  tests 
were  conducted  at  room  temperature  only. 

•  The  F-16  and  F/A-18  employ  random  vibration  at  the 
unit  (WRA/LRU)  level  whereas  the  F-15  does  not. 
(Implementation  not  available  when  F-15  production 
began,  next  generation  F-15  radars  will  include  random 
vibration. ) 

•  At  the  system  level,  the  F-16  set  does  not  receive 
burn-in  per  MIL-STD-781B  (temperature  cycling  with 
sine  vibration.)  Starting  in  January  1983,  a  self- 
imposed  five-day  sample  test  on  one  radar  per 
month  using  the  RQT  thermal  cycling  was  initiated. 

Each  F-15  and  F/A-18  set  receive  48  hours  and  35 
hours  of  burn-in,  respectively,  before  delivery. 

The  ESS  program  for  the  APG-65  F/A-18  radar  was  built 
on  experience  gained  on  the  APG-63  F-15  program.  The  conditions 
used  in  the  APG-65  ESS  have  evolved  considerably  from  those 
initially  imposed  at  the  beginning  of  the  program.  Evolution 
has  been  toward  shorter  minimum  screens  but  longer  "failure 
free"  intervals.  The  use  of  "failure  free"  cycle  requirements 
allows  shorter  screening  on  units  with  no  screening  failures  and 
imposes  additional  screening  time  on  units  that  are  experiencing 
failures.  F-15  and  F/A-18  experience  indicates  that  flexibility 
is  needed  in  establishing  screening  methods  and  specifications 
and  that  supplier  involvement  is  necessary. 
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The  FIREFINDER  screening  program  included  high-temperature 
tests  of  ICs  at  receiving  inspection  initially  with  the  program 
being  in  the  process  of  changing  the  requirement  to  tests  at  both 
high  and  low  temperature.  Selected  complex  equipments  were 
screened  at  the  supplier  or  in-houi-e  and  the  system  was  subjected 
to  high-temperature  burn-in  at  50°C  for  100  hours  with  the  last 
24  required  to  be  "failure  free"  (Ref.  FIREFINDER,  pp.  56/2-22). 
Additional  screening  using  temperature  cycling  and  low  temperature 
test  at  the  signal  processor  unit  level  is  being  developed. 

2.  Proaress  in  the  ESS  Area 


With  the  increasing  complexity  of  modern  electronic  systems, 
there  has  been  increasing  attention  to  the  area  of  environmental 
stress  screening.  There  have  been  a  number  of  recent  studies 
attempting  to  gather  the  best  information  on  this  subject  and  to 
provide  direction  and  guidance  to  the  engineer  or  manager  who  is 
attempting  to  structure  an  effective  environmental  stress 
screening  program.  In  the  course  of  the  study,  a  variety  of  these 
activities  was  reviewed.  The  most  comprehensive  of  these  efforts 
is  the  National  Program  on  Environmental  Stress  Screening  of 
Electronic  Hardware  ( ESSEH )  by  the  Institute  of  Environmental 
Sciences.  The  program  was  initiated  in  1979  to  attempt  to  bring 
some  order  and  consistency  to  ESS.  The  situation  at  that  time 
was  typified  by  most  of  the  industry  working  unilaterally  with 
the  results  of  these  efforts  being  unpublished  and  unshared. 

The  ESSEH  program  has  attempted  to  gather  this  experience  together 
and  to  make  a  usable  guideline  for  others  to  use.  The  initial 
IES  guideline  on  ESS  was  published  in  1981.  A  continuing  effort 
is  directed  at  updating  this  guideline  and  adding  information  on 
part  screening  and  updating  assembly  level  screening  guidelines. 

Among  the  other  recent  studies  on  this  subject  that  were 
examined  by  the  study  was  the  RADC  technical  report  ( RADC  TR-82- 
87),  Stress  Screening  of  Electronic  Hardware.  This  report  and 
the  work  currently  being  accomplished  on  RADC  contract  F-30602- 
82C-0121  will  serve  as  the  basis  for  a  draft  standard  on  ESS. 

ESS  cost  considerations  are  discussed  in  Appendix  F. 
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A  system  for  identification  and  correction  of  failure  modes/ 
problems  is  essential  to  design  maturation  and  reliability  growth. 

The  cases  make  a  strong  statement  that  a  closed  loop  failure 
reporting,  analysis  and  corrective  action  system,  emphasized  by 
program  management,  established  early  in  the  development  effort 
and  continued  through  production,  is  effective  and  provides  cost 
benefit.  The  earlier  in  the  system  life  the  problem  is  identi¬ 
fied  and  corrective  action  applied,  the  less  costly  in  dollars 
and  perturbation  to  the  system  it  is. 

The  FIREFINDER  case  study  makes  the  point  that  upper  manage¬ 
ment  participation  and  assignment  of  a  responsible  person  are 
keys  to  a  successful/effective  FRACAS.  For  FIREFINDER,  the 
Failure  Review  Board  ( FRB )  was  the  focal  point  for  reviewing  the 
status  of  all  failed  items  reported  on  Operation  and  Maintenance 
Reports  (OMRs),  field  failure  reports  and  other  reported  problems 
through  the  Responsible  Assigned  Engineer  (RAE). 

FIREFINDER  OMRs  were  written  by  systems  test  personnel  for 
all  operational  and/or  maintenance  discrepancies  occurring.  Failed 
hardware  is  dispositioned  by  the  RAE.  Preliminary  Review  Team  ( PRT ) 
reviews  all  OMRs,  field  failure  reports  or  other  failure  data,  and 
assigns  them  to  the  appropriate  RAE  for  resolution.  Factory  failure 
trend  data  are  compiled  by  Quality  and  the  Project  Offices  for  FRB 
review.  The  FRB  assigns  problems  to  the  RAE.  The  FRB  monitored  pro¬ 
gress  of  the  RAE  activity  in  regularly  scheduled  weekly  meetings, 
reviewed  corrective  actions  per  flow  diagrams  and  closed  items 
as  warranted.  Test  discipline  assures  that  information  required 
for  good  analysis  is  preserved.  Test  discipline  is  addressed 
in  test  procedures  in  strict  implementation  of  those  procedures, 
and  in  independent  review  of  failure  reports  to  assure  complete¬ 
ness  (Ref.  FIREFINDER  Case  Study,  p.  140).  A  key  aspect  of  the 
FIREFINDER  FRB  is  the  involvement  and  responsibility  of  the 
engineering  activity.  The  technical  director  is  the  Chairman 
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and  he  is  directly  responsible  to  management  for  FRACAS  effective¬ 
ness.  He  is  assisted  by  co-chairman  from  reliability  and  product 
assurance  (quality). 

The  F-15  and  F/A-18  radar  cases  made  reference  to  the  CLEAR 
system  as  a  valuable/effective  FRACAS.  The  cases  stated  that 
early  in  the  1970s,  MCAIR  developed  the  CLEAR  (Closed  Loop  Eval¬ 
uation  and  Reporting  System)  to  satisfy  F-15  program  needs  for  re¬ 
liability,  maintainability,  and  quality  engineering  information 
in  addition  to  being  complementary  with  the  AFM  66-1  system.  CLEAR 
integrated  into  one  system  what  had  previously  been  many  separate 
MCAIR  information  gathering  systems  to  provide  nonconformance, 
malfunction,  maintenance,  and  safety  data.  In  addition  to  com¬ 
bining  data  sources,  new  multipurpose  forms  were  designed  and 
used  to  input  information  more  efficiently  into  computer  systems, 
thereby  minimizing  manual  operations  and  providing  expanded  analy¬ 
sis  and  reporting  capabilities.  CLEAR  is  used  during  production, 
assembly,  laboratory  testing,  supplier  testing,  flight  testing 
and  initial  field  operations,  or  where  MCAIR  is  providing  repair 
and  support  services.  The  system  provides  computerized  outputs 
which  are  used  to  fulfill  F-15  contract  requirements  for  reli¬ 
ability,  maintainability,  and  quality  reporting.  The  case  stated 
that  the  three  major  building  blocks  of  CLEAR  (reporting,  input, 
and  computer  processing)  have  been  improved  steadily  with  time 
and  adapted  for  use  on  the  F/A-18  and  currently  the  AV-8B  (Ref. 
F/A-18  Case  Study,  pp.  166-168). 

An  F-16  Radar  Field  Performance  Evaluation  group  was  initiated 
in  1980  to  provide  R&M  field  performance  visibility  to  management 
and  engineering  and  to  identify  R&M  problem  areas.  An  R&M  com¬ 
puterized  data  base  has  been  generated  with  input  data  from 
contractor  field  engineers  and  the  RIW  data  base.  Problem  areas 
identified  by  an  analyzer  of  this  data  are  provided  to  the  "Problem 
Action  Team"  for  evaluation,  resolution  and  reporting  to  program 
management.  Periodic  R&M  performance  visibility  reports  are  also 
derived  from  the  data  base  for  distribution  to  management  and 
design  engineering. 
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E.  TEST  AND  EVALUATION 


The  F-16  radar  case  study  states  that  the  program  used  a  great 
many  different  kinds  of  tests  to  identify  and  correct  problem 
areas  and  verify  compliance  with  specified  requirements. 

The  tests  and  evaluations  were  grouped  into  four  categories: 
laboratory  tests  (development  and  production),  DT&E  flight  test 
(development),  OT&E  flight  test  (operational),  and  in-service 
assessment.  Development  laboratory  tests  included  Thermal 
Analysis  Verification,  Reliability  Growth,  Reliability  Quali¬ 
fication,  Maintainability  Demonstration  and  AIS  compatibility 
tests.  Production  tests  included  Reliability  Growth,  Reliability 
Qualification  and  Reliability  Acceptance  tests. 

The  F/A-18  grouped  test  and  evaluation  into:  development 
test,  reliability  development  test,  design  limit  qualification 
test,  reliability  demonstration,  initial  BIT  assessment,  main¬ 
tainability  demonstration,  flight  assessment,  and  in-service 
R&M  assessment  (Ref.  F/A-18  Case  Study,  p.  184). 

The  case  study  analysis  examines  six  general  areas.  The 
six  areas  are:  integrated  testing,  design  limit  (environmental) 
qualification  tests,  R&M  growth/maturity  testing,  demonstration 
testing,  operational  testing,  and  in-service  testing. 

1 .  Integrated  Testing 

The  cases  studied  indicated  that  most  tests  can  provide 
data  that  can  be  used  as  indicators  of  inherent  reliability  and 
for  discovery  of  pattern  failures/system  reliability  flaws.  The 
F/A-18  test  and  evaluation  program  was  purposely  integrated  and 
interleaved  with  many  of  the  individual  tests  building  on  one 
another.  All  (test)  failures  were  analyzed  and  followed  up  for 
necessary  corrective  action  (Ref.  F/A-18  Case  Study,  p.  185). 

Most  of  the  F-15  radar  LRUs  had  elapsed  time  indicators,  and 
early  radar  experience  came  from  periodic  readings  made  throughout 
the  development  program.  These  readings  and  the  test  data  from 
various  tests  could  be  used  to  predict  subsequent  reliability 
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(Ref.  F-15  Case  Study,  p.  IIE-2).  Results  of  many  tests  can  be 
indicators  of  expected  reliability  during  operation  usage.  The 
cases  point  up  the  fact  that  all  development  tests  are  sources 
of  R&M  data  for  failure  analysis,  R&M  indicators  and  corrective 
action.  The  developing  and  changing  configuration  and  the  test 
environments  may  not  replicate  the  final  configuration  enviro¬ 
nment  and  may  not  allow  a  direct  measure  of  final  system  R&M 
characteristics,  but  the  value  of  the  data  in  accessing/improving 
R&M  should  not  be  dismissed.  Modifications  to  the  various  devel¬ 
opment  tests  may  be  necessary  to  enhance  the  value  of  the  data 
for  use  in  R&M  evaluations  and  improvement,  or  vice  versa,  but 
the  value  can  be  seen  as  indicated  in  the  F/A-18  study.  The 
F/A-18  study  makes  a  point  throughout  the  case  that  all  test 
results  support  a  TAAF  effort,  not  just  a  dedicated  growth/TAAF 
test  phase  (Ref.  F/A-18  Case  Study,  p.  185). 

2.  Design  Limit  Qualification  Tests  (Environmental 
Qualification  Tests) 

The  AN/TPQ-36  Environmental  Testing  included  Qualification 
Testing  of  the  system  in  extreme  environments  during  engineering 
development.  Corrective  actions  instituted  to  correct  Qualifi¬ 
cation  Test  problems  and  enhancement  developed  using  Qualification 
Test  Results  contributed  to  the  improvement  in  R&M  characteristics 
during  the  engineering  development  phase.  Verification  of 
corrections  and  qualification  of  the  TPQ-36  design  was  provided 
by  the  First  Article  Test  Program.  Performance  in  extreme 
environments  was  also  tested  periodically  throughout  production 
as  a  part  of  program  testing  (Ref.  FIREFINDER  Case  Study,  p.  144). 

The  F/A-18  study  makes  a  point  that  to  minimize  the  proba¬ 
bility  of  retrofit,  high  stress  testing  must  be  performed  early 
in  the  program,  and  the  results  of  all  testing  must  be  used  in  a 
TAAF  (Reliability  Growth)  concept  (Ref.  F/A-18  Case  Study,  p.  188) 
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The  F/A-18  utilized  stress  tests  that  subjected  the  equipment 
to  low  and  high  thermal  extremes,  with  a  rapid  rate  of  change 
between  the  extremes,  to  reveal  any  weakness  related  to  the  high 
rates  of  change  in  temperature  (Ref.  F/A-18  Case  Study,  pp.  190-198). 

The  F-16  utilized  testing  during  development  and  during 
production  under  combined  environments.  These  included  random 
vibration,  rapid  temperature  excursions,  power  cycling  and  limited 
altitude  and  humidity  stresses.  The  testing  was  conducted  to 
provide  early  detection  and  correction  of  problem  areas  and  to 
enhance  compliance  with  specified  R&M  requirements.  Extensive 
use  was  made  of  random  vibration  and  temperature  profiles  that 
were  representative  of  actual  flight  conditions  during  demon¬ 
strations. 

The  Design  Limit  Qualification  test  on  the  LDNS  tests  was 
performed  in  accordance  with  the  test  procedures  of  MIL-STD-810, 

461,  and  704.  The  purpose  of  the  tests  was  to  determine  how  the 
equipment  would  operate  under  the  environmental  conditions 
imposed.  Failure  analyses  were  performed  on  failed  components 
and  verification  of  repairs  accomplished.  No  data  were  collected 
to  calculate  R&M  limits.  The  sequence  of  tests  was  arranged  such 
that  all  nondamaging  tests  were  performed  prior  to  damaging  tests. 

It  is  interesting  to  note  that  all  qualifications  tests  were 
completed  prior  to  Reliability  Demonstration  Test. 

3 .  R&M  Growth/Maturity  Testing 

Each  case  discusses  a  R&M  growth  phase.  The  F-16  and  TPQ-36 
had  dedicated/planned  R&M  growth  test  phases,  while  the  F-15  had 
specified  hardware  and  software  corrective  action  phases  that 
translated  to  growth  effort,  and  the  F/A-18  made  a  strong  point  that 
all  testing  was  in  fact  TAAF  testing  since  all  malfunctions  would 
be  analyzed  and  fixes  fed  back  into  the  system.  A  formal  reli¬ 
ability  growth  testing  was  not  part  of  the  LDNS  Engineering  Test 
Program;  however,  reliability  growth  was  accomplished  as  part  of 
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the  RIW  phase.  The  philosophy  of  growth  testing  was  evident  in 
each  case  and  R&M  growth  was  tracked  as  a  valuat le  management 
tool  in  each  case.  Growth  charts,  plans,  projections,  programs 
and  analyses  are  evident,  to  various  degrees,  in  each  case 
studied . 

a.  Reliability  Growth  Tests 

The  TPQ-36  reliability  growth  tests  were  conducted  late  in 
the  program  (during  the  production  phase).  The  objectives  of  the 
growth  tests  were  to  use  the  TAAF  principle  to  surface  and 
correct  deficiencies  in  design  parts  or  workmanship.  The  two 
"growth  radars"  were  operated  in  a  simulated  field  environment 
including  system  on-off  cycling,  Munson  road  travel  and  march 
order  emplacement.  The  two  systems  accumulated  2800  operating 
hours  (in  6  months  on  each)  and  200  non-operating  hours  over  the 
Munson  road  course.  All  potential  fixes  were  proofed  and  verified 
for  inclusion  into  production  units  (Ref.  FIREFINDER  Case  Study, 
pp.  152,  153). 

On  the  F-16  radar  program,  1500  test,  analyze  and  fix  hours 
were  expected  to  mature  the  FSD  model  to  achieve  the  60  hour  MTBF 
required  in  the  APG-66  Reliability  Qualification  Test  ( RQT ) . 
Following  the  1500  hours  growth  test,  and  RQT,  an  additional  500 
hour  dedicated  growth  test  (RGT)  was  planned.  The  2000  hours  of 
growth  testing  were  expected  to  mature  the  production  model  radar 
to  the  100  hour  MTBF  level.  Four  test  articles  were  planned  for 
the  RGT.  In  actuality  one  test  article  was  used  for  the  RGT  for 
a  total  of  420  hours  with  23  corrective  actions  taken. 

During  the  F-16  competition  for  a  radar  supplier,  it  became 
evident  that  the  TWT  was  a  high-risk  item  and  likely  to  be  a  cost 
driver.  A  reliability  test  requirement  for  the  TWT  was  devised 
and  included  in  the  RFP.  The  selected  radar  contractor  then 
further  imposed  a  reliability  growth  test  program  on  their  TWT 
contractors.  The  field  results  of  this  TWT  growth  program  were 
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judged  so  outstanding  that  the  next  generation  of  TWT  also  has  a 
similar  reliability  growth  program  specified.  The  successful 
design  currently  has  demonstrated  four  times  the  MTBF  predicted 
originally. 

The  F/A-18  radar  program  treated  all  tests  as  test,  analyze 
and  fix  (TAAF.)  However,  a  reliability  development  test  ( RDT) 
was  conducted  using  two  preproduction  radars  for  a  total  of  1592 
equipment  hours.  The  RDT  was  conducted  in  operationally  repre¬ 
sentative  environments.  Approximately  136  failures  were  en¬ 
countered  during  the  RDT.  The  RDT  was  judged  to  have  made  a 
significant  contribution  to  reliability  growth  although  its  tim¬ 
ing  was  such  that  many  of  the  problems  identified  had  been  found 
during  other  tests  (Ref.  F/A-18  Case  Study,  pp.  190-197). 

b.  Total  Program  for  Reliability  Growth 

The  case  studies  indicate  that  reliability  growth  planning 
was  important  to  the  reliability  levels  achieved  by  the  systems 
studied.  Growth  was  not  just  the  result  of  dedicated  TAAF  phases, 
but  must  include  failure  analyses,  corrective  action  determination 
for  every  test,  the  incorporation  and  verification  of  corrective 
actions,  and  application  to  developed  systems.  The  case  studies 
indicate  that  a  growth  program  is  valuable  from  design  through 
field  use.  The  strong  emphasis  in  the  F/A-18  case  study  on  the 
value  of  all  test  data  as  potential  growth  test  data  should  not 
go  unnoted.  The  fact  that  each  program  addressed  a  reliability 
growth  program,  and  indicated  management  attention  directed  to 
planned  and  achieved  reliability  goals/thresholds,  is  positive 
indication  of  the  value  successful  program  managers  place  on  a 
total  growth  program. 

The  case  studies  reveal  that  treating  every  failure/incident 
as  a  potential  opportunity  to  reduce  the  failure  rate  pays  big 
dividends.  The  F/A-18  radar  case  study  indicates  that  the  ongoing 
production  screening/burn-in  and  corrective  action  efforts  resulted 
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in  continued  reliability  growth  during  the  early  phases  of  fielding. 
The  early  F/A-18  radar  improvements  were  expedited  by  contractor 
field  teams  that  moved  from  base  to  base  to  incorporate  planned  cor¬ 
rective  action  modification. 

The  F-16  radar  case  study  indicates  that  production  reliability 
growth  testing,  reliability  qualification,  and  reliability 
acceptance  testing  as  well  as  burn-in/screening  tests  were  strong 
contributors  to  cor tinued  reliability  growth  in  production. 

Failures  were  analyzed  and  corrective  action  was  implemented  for 
problem  areas  identified  during  each  test  including  failures 
during  successfully  completed  reliability  qualification  and 
reliability  acceptance  tests. 

The  TPQ-37  experienced  an  abrupt  increase  in  reliability  em¬ 
phasis  when  the  lack  of  an  advanced  development  reliability  re¬ 
quirement  was  replaced  by  a  firm  90-hr  MTBF  requirement  for  low- 
rate  initial  production  (LRIP)  (Note:  there  was  no  ED  phase). 

Every  part  of  the  LRIP  effort  became  a  potential  input  to  a 
program  of  reliability  growth.  The  success  of  this  effort  is 
illustrated  by  the  increase  in  the  DT  I/OT  I  MTBF  from  24/45  hr, 
respectively,  to  87/94  hours  MTBF  in  DT  III/OT  III. 

A  more  complete  discussion  of  the  value  of  Reliability  Growth 
program  and  a  summary  of  how  to  structure  the  program  are  presented 
in  Section  IV-B  of  this  volume. 

c.  BIT  Maturation  Program 

The  case  studies  reinforce  the  concept  that  for  the  BIT 
design  to  mature,  BIT  failures  must  be  experienced  or,  at  least, 
the  BIT  must  see  equipment  failures.  As  an  example,  the  F-15 
case  displays  extensive  "software  corrective  actions"  resulting 
from  each  of  the  R&M  tests.  It  is  noted  that  a  large  portion  of 
the  corrective  actions  involve  BIT  enhancements  and  a  large 
portion  of  those  involve  software  changes  to  reduce  false  alarms. 

A  goal  of  BIT  maturation  is  to  reduce  false  alarms,  and  thus 
increase  confidence  in  BIT  (Ref.  F-15  Case  Study,  p.  IIE-27,  28). 
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A  large  amount  of  testing  provides  the  opportunity  for  ob¬ 
taining  data  on  hard  failures  to  exercise  the  BIT.  Hence,  hard 
failures  decrease.  It  is  important  to  obtain  data  in  all  testing 
to  help  mature  BIT.  As  the  hardware  reliability  improves,  it  is 
necessary  to  insert  faults  to  evaluate  growth  BIT.  Data  should  be 
obtained  and  evaluated  on  all  tests  to  minimize  false  alarm 
problems.  See  Section  IV-C  for  an  extensive  discussion  on  BIT 
diagnostic  growth. 

The  F/A-18  initial  BIT  assessment  conducted  in  May,  1980 
was  an  early  hardware  and  software  evaluation  of  the  supplier's 
BIT  design.  These  tests  were  conducted  prior  to  the  reliability 
development  test.  The  BIT  isolated  approximately  77  percent  of 
the  faults  inserted.  This  early  test  helped  to  focus  the  matura¬ 
tion  effort. 

The  formal  F/A-18  BIT  test,  to  be  conducted  later  in  1983,  will 
use  randomly  selected  faults  proportionally  distributed  in  accordance 
with  WRA  failure  rates,  to  test  the  maturity  of  the  BIT.  The  test 
accept  criteria  is  quite  stringent  (0  test  failure  allowed  for  95 
faults,  one  test  failure  will  require  30  additional  faults  to  be 
inserted  with  test  failure,  and  so  on)  (Ref.  F/A-18  Case  study, 
pp.  202,  203,  207). 

The  TPQ-36/37  emphasized  early  maturation  of  BIT  by  design 
and  test  of  BIT  along  with  card  design.  The  contractor  attempted 
to  test  every  failure  mode  and  failure  location.  The  plan  was 
very  effective  and  showed  maturity  very  early  in  the  program.  BIT 
maturity  and  confidence  rose  early  enough  in  the  program  that  BIT 
was  used  as  the  indicator  in  the  reliability  demonstration  tests. 

F-16  radar  BIT  improvements  were  made  at  ECP  331,  block 
change  point.  Several  changes  were  made  to  the  self-test  and  BIT. 

The  improvements  were  made  to  enhance  pilot  and  maintainer 
confidence  and  ease  of  use  (Ref.  F-16  Case  Study,  p.  IIE-62). 

A  more  complete  discussion  of  maintainability  growth  program 
is  presented  in  Appendix  E. 
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4.  Demonstration  Testin' 


Demonstration  testing,  properly  specified  and  implemented, 
can  be  a  powerful  tool  for  assuring  reliability  growth  to  specified 
levels  and  providing  leverage  and  engineering  data  for  incorpora¬ 
tion  and  evaluation  of  corrective  action.  Each  case  study  con¬ 
tains  charts  of  test  results  that  may  be  consulted  for  detail. 
Figure  C-20  is  a  summary  of  the  type  and  depth  of  the  demonstra¬ 
tions. 

In  the  case  of  the  APG-66  radar,  Reliability  Qualification 
Tests  ( RQT)  and  Reliability  Acceptance  Tests  (RAT)  were  judged  by 
the  contractors  involved  to  be  among  the  major  reasons  the  radar 
has  done  so  well  in  terms  of  field  reliabiliity.  The  APG-66 
reliability  test  program  included  development  growth  testing 
followed  by  development  RQT  and  production  growth  testing  followed 
by  production  RQT  and  RAT.  Each  test  resulted  in  incorporation 
of  extensive  corrective  action.  In  addition,  demonstration 
tests  resulted  in  retest  to  evaluate  effectiveness  of  corrective 
action  and  demonstrate  compliance  with  required  levels  of  reli¬ 
ability.  Corrective  action  was  identified  and  incorporated  for 
all  failures  including  those  that  occurred  in  demonstration 
tests  that  satisfied  the  accept/reject  criteria.  Production  RQT 
and  RAT  are  effective  in  identifying  production  problems  not 
relatable  to  development  and  verifying  their  fixes. 

The  F-16  radar  program  attempted  to  hold  to  schedule  (the 
only  slip  was  a  6-8  month  FSD  RQT  slip  due  to  hardware  unavail¬ 
ability).  A  FSD  pre-RQT  was  performed  which  was  expected  to 
improve  the  probability  of  passing  RQT.  The  system,  at  the 
completion  of  FSD  growth  testing,  was  submitted  to  the  FSD  Reli¬ 
ability  Qualification  Test.  The  system  failed  in  the  first 
attempt,  corrective  actions  were  implemented  and  it  passed  in  the 
second  attempt.  The  production  RQT  required  three  attempts 
before  the  system  successfully  passed  the  test.  The  production 
system  was  submitted  to  a  production  acceptance  test  and  was 
rejected  on  the  first  two  attempts,  and  was  passed  on  the  third 
attempt. 
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The  F-15  case  shewed  progressive  MTBF  growth  thresholds  (30, 

45,  60)  with  demonstrations  required  to  show  attainment  of  the 
required  thresholds.  The  case  goes  to  considerable  length  to 
show  the  relationships  of  known/firm  requirements,  and  demonstrations 
to  building  confidence  in  meeting  user  thresholds.  The  F-15 
approach  of  creating  a  series  of  demonstrations,  encouraging 
Failure  Analysis  and  Corrective  Action,  and  application  to  follow- 
on  equipment  and  further  demonstrat ion ,  supports  the  value  of 
demonstration  testing. 

The  first  30-hr  MTBF  RQT  was  started  in  November  1973,  14 
months  later  than  originally  planned,  one  year  after  Environmental 
Qualif ication  Tests  were  initiated,  and  four  months  before  they  were 
completed.  Factors  also  contributing  to  the  delay  in  the  start 
date  were  reassignment  of  RQT  assets  and  difficulties  with  test 
chamber  facilities.  During  early  attempts  to  start,  test  facilities 
were  less  reliable  than  the  radar  being  tested. 

Planned  start  dates  for  the  production  reliability  tests 
( PRT )  covering  the  45-hr  and  60-hr  test  were  also  delayed  by  15 
months  or  more.  These  test  schedule  slippages  were  due,  in  part, 
to  having  only  one  test  facility  which  became  available  only  when 
each  test  was  completed.  Many  of  the  delays  involved  the  solu¬ 
tion  to  software  problems  which  held  up  conducting  the  test. 

Once  the  demonstration  tests  began  they  were  successful.  As  of 
December  1982,  no  APG-63  radar  had  failed  to  pass  its  required 
reliability  test  (Ref.  F-15  Case  Study,  pp.  IIE-10  to  IIE-52). 

The  F-15  radar  case  makes  the  statement:  "It  is  inherent  in 
MIL-STD-781  demonstration  testing,  which  places  a  premium  on 
measured  MTBFs ,  that  prime  contractors  and  subcontractors  will 
be  concerned  with  passing  tests.  Since  there  is  a  financial 
cost  and  a  matter  of  reputation  associated  with  failure,  it 
follows  that  test  delays  tend  to  be  the  result  since  these  do 
not  involve  incentives  or  substantial  penalties"  (Ref.  F-15  Case 
Study,  p.  I IE-11 ) . 
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The  TPQ-37  program  began  with  no  quantitative  reliability 
requirement  and  therefore  no  contractor  demonstration,  just  a  MTBF 
goal  of  250  hours.  A  firm  MTBF  requirement  of  90  hours  with  a 
demonstration  test  was  added  to  the  LRIP  contract.  After  incorporation 
of  the  demonstration  requirement,  the  TPQ-37  experienced  an  abrupt 
increase  in  reliability  emphasis  and  every  part  of  the  LRIP  effort 
became  potential  input  to  a  program  to  grow  reliability.  The 
demonstration  requirement  combined  with  an  incentive  provision 
resulted  in  growth  from  DTI/OT  I  MTBF  of  24/45  hours,  respectively, 
to  an  MTBF  of  125  hours  (Ref.  FIREFINDER  Case  Study,  pp.  164,  165). 

The  F/A-18  case  study  indicates  that  the  85-hour  MTBF  requirement 
(that  for  later  units  rose  to  106-hour  MTBF  requirement)  was  in 
the  minds  of  the  program  manager  and  motivated  the  development 
effort  to  produce  a  "106-hour  radar,"  as  was  the  M/BIT  Demo 
requirement. 

A  Reliability  Demonstration  Test  was  performed  on  the  LDNS 
during  the  development  program  to  determine  that  the  equipment 
design  complied  with  the  reliability  requirement  in  terms  of  the 
specified  MTBF  of  1000  hours.  The  test  was  performed  in  accordance 
with  Plan  XXI  of  MIL-STD-781B.  The  total  test  time  (equipment 
on)  was  1840  hours  with  three  failures  occurring.  This  test  was 
the  only  measure  of  acceptance  for  the  reliability  requirement. 

A  point  is  made  in  each  case  that  very  often  insufficient 
assets  are  available  during  the  development  and  preproduction 
phases.  When  this  happens,  engineering/growth/test  events  are 
strung  out,  valuable  data  are  delayed  and  assets  are  sometimes 
submitted  to  conflicting  environments  (e.g.,  reliability  tests  and 
destructive/degrading  tests  using  the  same  sample/item).  Certain 
R&M  development  tests  are  cut  out  because  the  R&M  sample  is 
diverted  to  another  purpose. 

The  cases  studied  indicate  that,  when  properly  applied,  demon¬ 
stration  testing  is  a  motivator  to  the  contractor,  and  a  useful 
tool  for  the  procuring  agency  to  gain  assurance  that  the  desired 
characteristics  contracted  for  are  inherent  in  the  item. 
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R&M  characteristics  are  not  as  obvious  in  a  system  as  are 
other  characteristics  like  weight,  power,  size,  speed,  etc.  Reli¬ 
ability  characteristics  and  to  a  lesser  extent  maintainability 
characteristics  are  dependent  on  factors  such  as  the  operational 
modes,  mission  profiles,  environments  and,  of  course,  time. 

The  demonstration  is  a  structured  test  in  a  controlled 
environment.  The  reliability  or  maintainability  demonstration 
appears  to  be  the  best  measure  of  the  true  reliability  inherent 
in  the  system,  other  than  the  field  use.  The  demonstration  is, 
of  course,  the  event  at  which  government  and  management  determine 
pass/fail,  acceptability  of  design,  LCC  considerations,  incentive 
payments,  progress  toward  awards,  and  meeting  established  R&M 
thresholds. 

A  formal  agreed-upon  set  of  characteristics  demonstrated  in 
accordance  with  agreed-on  definitions,  conditions  and  standards 
appears  to  be  supported  by  each  study  as  desirable  and  essential. 

5.  Operational  Testing 

Each  case  study  discussed  operational  test  in  some  form 
and  emphasized  the  need  to  satisfy  the  user  of  the  system.  The 
cases  report  that  every  attempt  must  be  made  to  bring  the  system 
up  to  or  above  the  original  requirement  of  the  ultimate  user. 
"Operational  Testing"  has  many  names  (MTO&E,  OT,  Flight  Testing, 
etc.),  but  each  case  emphasizes  that  the  operational  environment 
is  "final  true  test"  of  the  developed/produced  system  and  new 
problems  will  be  discovered  which  must  be  fed  back  into  the 
system  for  corrective  action. 

6.  In-Service  R&M  Assessment 

The  ultimate  demonstration  of  system  R&M  characteristics  and 
their  impact  on  readiness,  mission  capability,  operating  and 
maintenance  costs,  is  after  the  introduction  of  the  system  into 
the  operational  units/force.  Each  of  the  cases  presented 
included,  in  some  form,  a  system  for  field  R&M  assessment  and 
feedback  into  the  corrective  action/growth/maturation  cycle. 
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The  F-16  radar  included  an  Riw  contract  with  a  specific 
turnaround  time  requirement.  The  F-16  case  expressed  the  program's 
success  measures  in  relation  to  field  results  attained,  such  as 
MFHBF  of  65  hr,  lower  O&S  cost,  achieving  predicted  mature 
maintainability,  fault-free  radar  flights,  and  an  operational 
readiness  exceeding  98  percent  FMC.  (Ref.  F-16  Case  Study,  p.  ID-3.) 
The  LDNS  was  a  RIW  contract  and  reliability  improvements  similar 
to  those  experienced  on  the  F-16  occurred  after  the  systems  were 
deployed. 

The  F/A-18  radar  case  study  described  the  field  reliability 
increasing  throughout  the  period  of  1980  to  June  1982  to  December 
1982.  The  case  study  reports  mean  radar  hours  between  removals 
as  14,  19  and  24,  radar  repairs  as  27,  29  and  41,  and  primary 
failures  as  42,  48,  and  59  for  the  1981,  June  1982  and  December 
1982  reporting  points,  respectively.  The  case  described  in-service 
failure  assessment  as  critical  to  R&M  growth. 

The  F-15  case  included  extensive  field  reliability  data, 
and  credits  the  field  data  collection,  failure  analysis  and 
corrective  action/ECP  system  for  the  early  and  continuing  growth 
of  F-15  radar  reliability. 

The  F-15  case  makes  a  strong  statement  that  comparing 
specification  MTBF  with  field  operational  ( AFM  66-1)  reliability 
is  expected  to  provide  a  3-to-l  ratio.  This  serves  to  underscore 
the  ever-present  problem  of  comparing  operational  R&M  require¬ 
ments/measures/results  with  contractual  specification  require¬ 
ments/measures/results  (Ref.  F-15  Case  Study,  p.  IIE-56). 

TPQ-36/37  mentions  the  fact  that  an  attempt  was  made  in 
those  systems  to  create  a  common  FD/SC  and  data  collection 
technique.  The  FIREFINDER  case  study  also  brought  out  that  the 
FIREFINDER  Program  Manager  provided  for  field  data  collection  and 
expects  R&M  improvement  during  the  deployment  phase  of  the  system 
life  cycle  to  be  accelerated  as  a  result  (Ref.  FIREFINDER  Case 
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The  FIREFINDER  field  data  program  was  initiated  in  February/ 
March/  1983/  with  trips  to  Fort  Lewis,  Washington,  and  Fort  Hood, 
Texas.  Preliminary  field  MTBFs  have  been  established  and  changes 
identified  in  operational  and  maintenance  procedures  and  technical 
manuals  to  improve  the  field  R&M  characteristics. 
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MECHANICAL  WEAPON  SYSTEMS  ANALYSIS 


Observations  and  analysis  contained  in  this  section  are  based 
on  a  review  of  the  OSD/IDA  R&M  Case  Study  report  for  the  T700  Engine 
(IDA  Record  Document  D-22)  and  various  other  inputs  on  mechanical 
systems  obtained  as  a  result  of  presentations  made  to  the  R&M 
Core  Group.  The  structuring  in  this  section  was  derived  from 
engine  programs  and,  while  it  might  be  representative  of  a  class 
of  mechanical  systems,  it  does  not  necessarily  encompass  all 
types  of  mechanical  systems.  For  this  analysis,  the  R&M  program 
activities  were  divided  into  six  major  categories.  They  are: 
contracting,  management,  design,  production,  test  and  evaluation 
and  observations.  These  six  categories  have  been  further  subdivided 
into  various  elements  similar  to  Appendix  C. 

A.  CONTRACTING 

The  General  Electric  T700-GE-700  Gas  Turbine  Engine  was 
developed  under  Contract  #DAAJ01-72-C-0381  (52)  with  the  U.S. 

Army  Aviation  Systems  Command,  dated  15  March  1972.  The  engine 
is  the  main  propulsion  system  for  the  Army's  newest  utility 
helicopter,  the  Sikorsky-built  UH-60A  Blackhawk,  with  derivative 
models  powering  several  other  helicopters  such  as  the  AH-64 
Apache  and  SH-60  Seahawk  (Ref.  T700  Case  Study,  p.  1-2). 

Line  items  in  the  contract  specifically  called  out  that  a 
Reliability  Program  and  a  Maintainability  Program  be  conducted 
in  accordance  with  previously  submitted  program  plans  which  had 
been  reviewed  and  coordinated  with  U.S.  Army  planners  well  before 
award  of  the  development  contract  (Ref.  T700  Case  Study,  p.  IIA-0). 
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1.  R&M  Requirements 
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Mechanical  systems  that  are  to  meet  successful  levels  of  R&M 
require  explicit  definition  in  contractual  documentation.  Develop¬ 
ment  of  requirements  for  R&M  can  range  from  technology  expressed 
as  MTBP ,  MTTR,  Shop  Visit  Rate  (SVR),  etc.,  to  specifics,  such  as 
operating  life  of  parts  not  easily  accessible,  level  of  skill 
required  for  maintenance  actions,  and  types  and  numbers  of 
tools  required.  Accomplishing  adequate  and  understandable 
requirements  is  a  function  of  the  system  and  its  defined  mission. 

In  both  the  Prime  Item  Development  Specification  (PIDS), 
and  in  the  separate  R&M  Program  Plans,  both  quantitative  and 
qualitative  requirements  were  specified  which  were  contractual 
requirements  (not  goals)  to  be  demonstrated  by  the  end  of  the 
development  contract.  They  were: 


Reliability  Requirements:  The  engine  shall  achieve 
the  specified  reliability  value  of  1200  hours  Specified  Mean-Time- 
Between-Failure  based  upon  decision  risks  of  10  percent  and  a 
discrimination  ratio  of  two  to  one  (Ref.  T700  Case  Study,  p.  IIA-16) 


Engine  Design  Life  Requirements;  The  engine  shall  have 
a  design  life  of  5,000  hours,  with  an  initial  target  of  1,500 
engine  operating  hours  MTBFRO  (Mean-Time  Between  Failure  Requiring 
Overhaul)  at  completion  of  the  Post  Qualification  Reliability 
Demonstration  Test  Program  (Ref.  T700  Case  Study,  p.  IIA-16). 


Maintainability  Requirements: 


•  Corrective  Maintenance  -  field  levels  -  .07  mhr./op.hr, 


•  Preventive  Maintenance  -  field  levels  -  .03  mhr./op.hr, 
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•  Total  Direct  Maintenance  -  all  levels  -  .24  mhr./op.hr. 

•  Mean-Time-Between-Maintenance  (MTBM )  -  220  engine  operat¬ 
ing  hours  (excluding  daily  inspection). 

•  Mean-Down-Time  -  field  levels  -  1.7  hours. 

•  Active  Elapsed  Time  to  repair  a  Class  V  failure  -  3  hours 
repair  or  servicing. 

•  All  organizational  repair  or  servicing  maintenance 
-  30  minutes. 

•  All  organizational  and  direct  support  maintenance  pro¬ 
cedures  shall  be  capable  of  performance  in  Arctic  cloth¬ 
ing  at  -54°C  without  degrading  the  mean  elapsed  time  by 
more  than  50  percent. 

•  The  remove/replace,  total  and  elapsed  times  and  special 
tool  requirements  were  presented  in  the  proposal  and  sub¬ 
sequent  specification  for: 

All  Modules  (4) 

All  Line  Replaceable  units  (LRU's)  (19) 

Power  Turbine  Module  Components  (10) 

(Ref.  T700  Case  Study,  p.  IIA-10) 

a.  R&M  Contract  Goals  Versus  Requirements.  Unless  as  high 
a  priority  is  given  to  R&M  as  is  given  to  the  other  factors  of 
contracted  requirements,  a  satisfactory  R&M  product  will  very 
likely  not  be  achieved.  The  case  studies  demonstrated  this 


most  clearly.  Hard,  well-defined  requirements,  rather  than 
goals,  are  important.  Requirements  should  be  expressed  in 
terms  that  can  be  translated  to  design  factors  such  as  accessi¬ 
bility,  specific  operating  lifetimes  of  key  components  known  to  be 
high  failure,  and  high  cost  items,  in  defining  requirements  the 
test  of  the  user  is  the  area  of  greatest  challenge  and  must  be 
dealt  with  carefully. 

b.  Defining  R&M  Requirements.  To  define  the  requirements, 
the  needs  of  the  customer  must  be  well-understood.  Lessons 
learned  in  Vietnam  were  used  as  a  basis  for  setting  R&M  require¬ 
ments.  This  established  a  need  for  significantly  improved 
engine  reliability  and  easier  flight  line  maintainability. 

Examples  of  requirements  not  previously  specified  in  engine 
developments  were:  (a)  inclusion  of  an  inlet  air  particle 
separator  as  an  integral  part  of  the  engine  (for  protection 
against  sand  and  dust  damage)  and  (b)  a  low-cycle  fatigue  test 
requirement  to  ensure  long  life  in  the  cyclic  nature  of  helicopter 
engine  operation. 

2.  Mission  Profiles 


To  establish  the  life  requirements  of  an  aircraft  gas  tur¬ 
bine  engine  such  as  the  T700,  it  is  necessary  to  define  the 
predicted  mission  usage  in  terms  of  percent  of  operating  time  at 
various  power  settings  (stress  rupture  life)  and  the  required  low- 
cycle  fatigue  life.  With  these  requirements  established,  the 
design  engineer  can  then  define  his  or  her  assigned  component 
design  to  meet  these  criteria.  As  easy  as  this  may  sound,  defining 
realistic  mission  requirements  in  advance  of  actually  fielding 
the  system  is  a  very  difficult  task.  If  the  time  at  maximum 
power,  for  example,  is  overstated  significantly,  then  parts  may 
be  overdesigned  which  can  affect  cost  and  weight.  If,  on  the 


other  hand,  mission  requirements  are  underestimated  in  the 
PIDS,  certain  parts  might  fall  short  of  meeting  the  stated  overall 
life  requirements  which  could  require  costly  redesign  at  a  future 
point  in  time. 

In  the  PIDS,  the  Engine  Design  Life  Requirement  was  defined 
in  the  classic  terms  of  percent  time  at  a  designated  power  level. 
This  in  effect  defined  a  mission  usage  profile  and  was  the 
original  basis  for  designing  the  various  engine  components  to 
meet  a  5000-hour  minimum  life.  This  power  spectrum  profile  coupled 
with  the  low-cycle  fatigue  requirement  of  15,000  cycles,  provided 
the  original  mission  profile  for  the  T700-GE-700  engine  design  (Ref. 
T700  Case  Study,  p.  IIA-22). 

3.  R&M  Failure  Definition 

The  failure  definitions  used  were  not  far  different  from 
those  used  in  other  case  studies.  The  excluded  failure  cate¬ 
gories  were: 

(a) .  Failures  resulting  from  errors  of  maintenance  per¬ 

sonnel. 

(b) .  Failures  resulting  from  operating  the  engine  beyond 

specification  limits,  included  failures  are  those 
operationally  related  failures  for  which  engine  provides 
integral  protective  devices  (overspeed,  overtemperature, 
hot  starts). 

(c) .  Failures  resulting  from  airframe  components. 

(d) .  Failures  to  start,  if  a  successful  start  is  accomplished 

without  corrective  maintenance  action. 
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Class  IV  -  Failures  which  result  in  power  loss  or  no 
start. 

Class  V  -  Failure  which  requires  unscheduled  maintenance. 
(Ref.  T700  Case  Study,  p.  IIA-A) 

4 .  Incentives 

R&M  contract  incentives  can  aid  in  developing  reliable  and 
maintainable  systems  by  focusing  contractor  management  atten¬ 
tion  on  actions  that  improve  and  ultimately  meet  or  exceed 
specified  requirements.  The  T700  program  contained  incentives 
that  were  successfully  implemented  to  the  contractor's  and  the 
government's  benefit.  The  RIW  (Reliability  Improvement  Warranty) 
played  an  important  role  in  this  program.  During  the  first 
three  years  of  production.  General  Electric  produced  the  T700 
under  a  warranty  incentive  agreement.  The  warranty  agreement 
broke  down  into  three  segments.  GE  had  to  absorb  100  percent 
of  the  cost  of  repairing  engines  or  components  that  failed 
during  the  first  250  operating  hours;  GE  and  the  Army  shared 
the  repair  costs  for  failures  between  250  and  500  operating 
hours.  GE  incentives  involved  cost  avoidance  up  to  500  hrs 
between  unscheduled  maintenance  with  a  positive  cash  flow 
increasing  linearly  from  500  to  750  hrs.  (See  Fig.  D-l ) 

It  obviously  was  very  much  in  GE' s  interest  to  produce  problem- 
free  T700  engines. 

Early  production  or  start-up  problems  turn  up  in  every  new 
production  engine  and  the  T700  was  no  exception.  A  couple  of 
problems  developed  which  had  to  be  corrected,  the  most  significant 
being  the  number  four  bearing  support  bottoming  in  the  midframe, 
thereby  negating  the  effect  of  the  bearing  support  oil  film 
damping  feature.  The  warranty  provisions  of  the  contract  enabled 


GE  to  move  quickly  and  successfully  to  solve  the  problem.  Approx¬ 
imately  90  percent  of  the  shop  visits  in  1979  were  removals 
to  correct  this  discrepancy  plus  one  or  two  other  start-up  problems 
It  is  believed  that  the  warranty  incentive  provided  a 
big  payoff  for  the  Army  because  it  hastened  early  solution  to 
start-up  production  problems  and  accelerated  achieving  a  mature 
engine.  The  Army  elected  to  discontinue  the  warranty  after 
three  years  of  production  (Ref.  T700  Case  Study,  p.  II.  C-105). 
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FIGURE  D-l .  T700  Warranty  Incentive 
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Source  Selection  Criteria 


In  the  cover  letter  of  the  Official  Request  for  Quote  ( RFQ ) , 
#DAAJ01-71-00455 ( 52 ) ,  which  was  issued  to  General  Electric,  Pratt 
and  Whitney  and  Lycoming  in  July,  1971  for  the  development  of  a 
1500-shaft-horsepower  turbine  engine,  the  following  statement 
appeared  in  the  block  entitled,  "ITEM(S)  TO  BE  PURCHASED  (Brief 
Description ) . 

"Design,  develop,  fabricate,  test,  demonstrate  reliability 
and  maintainability  and  qualify  a  1500-shaft-horsepower,  non- 
regenerative ,  direct  front  drive,  turboshaft  aircraft  gas 
turbine  engine  for  the  Utility  Tactical  Transport  Aircraft 
System  (UTTAS)." 

The  theme  of  this  brief  description  was  carried  throughout 
the  entire  RFQ,  leaving  no  doubt  that  the  U.S.  Army  was  very 
serious  about  R&M  being  of  prime  importance  on  this  new  engine. 

Under  Section  D  of  the  RFQ  the  Evaluation  Factors  were 
defined  for  making  the  award  of  a  contract  as  a  result  of  this 
RFQ.  In  paragraph  D.5,  three  major  evaluation  elements  were 
delineated  with  the  possible  points  to  be  awarded  for  each  of 
these  elements  as  follows: 


POSSIBLE  POINTS 


EVALUATION  ITEM 
Technical 
Management 
Cost 


Under  sub-paragraph  D5 . 1  Technical ,  sub-paragraph  D5.1.1 
breaks  down  the  elements  for  the  evaluation  of  the  design  and 
performance  of  the  engineering  and  logistical  critical  components 
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This  statement  points  out  that  specific  attention  should  be  given 
to  the  following  items: 


1)  Systems  Design 

2)  Component  Design 

3)  Trade-Off  Analysis 

Items  of  the  System  Design  will  include: 

1)  Sub-system  Development 

2)  Configuration/Weight  Analysis 

3)  Performance/Power  Extraction 

4)  Operating  Limitation 

5)  Reliability  and  Maintainability 

6)  Systems  Integration 

7)  Materials 

8)  Vulnerability  and  Serviceability 

9)  Producibility/Production  Margins 

10 )  Condition  Monitoring 

11 )  Diagnostics 

It  may  be  noted  from  this  set  of  evaluation  criteria  that 
Reliability  and  Maintainability  were  given  careful  consideration 
with  other  engine  characteristics  such  as  performance  and  weight. 
There  has  never  been  any  question  from  the  very  beginning  of  this 
program  that  Reliability  and  Maintainability  were  given  very  high 
priority  in  the  selection  of  this  new  Army  helicopter  engine  (Ref. 
T700  Case  Study,  p.  IIA-70). 
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B.  MANAGEMENT 

Because  success  in  R&M  for  programs  with  mechanical  systems 
involves  assurance  that  R&M  factors  carry  equal  billing  with  the 
other  program  elements,  it  is  necessary  that  management  overview 
of  k&M  be  emphasized.  In  the  T700  program,  maintainability  and 
reliability  were  given  the  highest  consideration,  and  this  priority 
was  clearly  conveyed  to  the  competing  contractors.  As  a  result 
of  this  emphasis,  the  correct  levels  of  planning,  control  and 
emphasis  were  established  at  the  prime  level,  subcontract  level, 
and  vendor  level. 

1.  Plannina.  Control  and  Bnphasis 


The  organizational  structure  is  the  key  to  meeting  the  plan¬ 
ning  control  and  emphasis  of  R&M.  R&M  cannot  be  treated  as  a  minor 
element  on  a  program.  The  case  studies  conducted  strongly  support 
the  idea  of  a  separate  R&M  manager  and  R&M  support  personnel 
integrated  with  engineering,  configuration  control,  etc.  The 
relationship  with  the  other  elements  must  obviously  be  very 
intimate,  and  adversary  relationships  have  to  be  avoided.  it  is 
the  responsibility  of  the  program  manager  to  see  to  this  since  a 
breakdown  here  will  destroy  the  probability  of  a  successful  R&M 
effort.  In  the  T700  engine  RFQ  the  Army  specified  that  the  R&M, 
Safety  and  Human  Factors  Engineering  Manager  report  to  the  Project 
General  Manager  at  the  same  reporting  level  as  the  Design  and 
ILS  managers  (Ref.  T700  Case  Study,  p.  IIB-4).  Strong  and  under¬ 
standable  planning  documents  must  be  prepared  and  adhered  to. 
Coordinated  efforts  must  be  orchestrated  properly.  Milestone 
charts,  PERT  charts,  road  maps,  or  whatever  one  may  wish  to  call 
them  need  to  be  used  to  ensure  that  R&M  effort  maintains  a 
course  to  successful  integration. 
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2 .  Monitoring  and  Control  of  Subcontractors  and  Suppliers 


As  important  as  the  above  is  to  the  prime  contractor's  efforts, 
the  same  concept  must  be  applied  to  subcontractors  and  vendors. 
Monitoring  and  control  efforts  must  be  established  to  accomplish 
these  tasks.  The  key  factors  to  accomplish  this,  found  in  the 
case  studies,  included  R&M  factors  clearly  spelled  out  in  RFQs, 
clear  and  precise  detailing  of  R&M  factors  in  program  plans, 
contractual  requirements  agreed  to  and  understood  between  prime 
and  sub,  and  equal  emphasis  applied.  Monitoring  required  proper 
reporting  to  the  prime,  in-depth  reviews,  and  adequate  screening 
of  sub-supplied  hardware  (Ref.  T700  Case  Study,  p.  IIB-22). 

3.  Engineering  Change  System 

It  is  important  to  avoid  delays  in  the  incorporation  of  ECPs 
found  necessary  to  ensure  good  R&M,  and  to  review  other  changes  to 
make  sure  R&M  has  been  properly  addressed.  The  review  and  develop¬ 
ment  of  change  documentation  must  be  done  with  the  same  care  as 
the  initial  program  when  being  contracted.  Unnecessary  delays  in 
the  approval  cycle  must  be  avoided  and  the  incorporation  must  be 
such  that  the  disruption  to  the  progress  of  the  program  is  held 
to  a  minimum. 

The  T700  case  study  verified  this  concern  and  demonstrated 
its  importance.  In  the  maintainability  and  mechanical  areas,  one 
of  the  key  elements  was  flexibility  in  processing  changes.  As 
stated  in  a  GE  briefing,  the  T700  Engine  Success  Story,  "the  ECP 
process  was  as  flexible  as  it  could  be,  which  allowed  the  con¬ 
tractor  to  recognize  and  correct  problems  as  quickly  as  possible." 
During  the  first  three  years  of  production  of  the  T700  helicopter 
engine,  General  Electric  was  under  contract  to  provide  total 
contract  support  which  provided  the  contractor  with  full  configu¬ 
ration  control  and  logistics  support  flexibility.  This  contract 
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provision  and  a  reliability  improvement  warranty  agreement  gave 
General  Electric  the  latitude  and  incentive  to  rapidly  correct 
R&M  problems  and  introduce  fixes  throughout  the  fleet.  Planning 
and  control  and  monitoring  played  a  large  and  important  role 
in  this  area. 

It  should  also  be  noted  that  relatively  few  ECP's  have  been 
necessary  for  the  T700,  a  fact  that  many  believe  to  be  the 
result  of  a  successful  maturity  period  after  completion  of  FSD 
and  prior  to  production. 

C.  DESIGN 

The  fact  that  an  engine  has  a  very  large  number  of  failure 
modes  due  to  the  nature  of  the  hostile  operating  environment  and 
the  drive  to  design  lightweight  efficient  structures  requires 
particular  attention  to  R&M  in  the  design  process.  Thermal  and 
mechanical  stresses  are  very  high  and  minimal  weight  is  a  para¬ 
mount  design  consideration.  The  aerodynamic,  mechanical,  struc¬ 
tural,  combustion/heat  transfer,  fuels/lubricants,  vibratory, 
controls,  etc.,  disciplines  are  severely  challenged  in  this  design 
process.  With  increasing  application  of  electronics  in  the 
engine  controls  area,  a  significant  challenge  exists  there  too, 
in  order  to  ensure  reliable  operation  of  electronics  in  a  high 
temperature,  noise,  vibratory,  and  Mg"  loading  environment. 

This  section  presents  these  issues  as  derived  from  the  General 
Electric  T700  Case  Study  and  the  USAF's  Engine  Structural  in¬ 
tegrity  Program  as  applied  to  the  F100  and  F101  engines. 
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1.  Development  of  Design  Requirements 

Achievement  of  a  truly  reliable  and  maintainable  engine 
required  a  consistent  high  priority  approach  from  the  Army  and 
General  Electric,  using  the  lessons  learned  from  past  experience. 
Fig.  D-2  summarizes  the  Army's  late-1960  Engine  Experience. 

U.S.  Army  Concept  Formulation  studies  for  replacement  of 
the  UH-1  transport  helicopter  began  in  the  mid-sixties  and 
resulted  in  a  system  called  UTTAS  (Utility  Tactical  Transport 
Aircraft  System).  This  system  had  some  challenging  requirements 
such  as: 

•  37%  -  50%  reduced  maintenance  man  hours 

•  20%  -  30%  reduced  fuel  consumption/engine 

•  Improved  survivabililty 

•  1,500  shaft  horsepower 

•  Integral  engine  protection  against  sand  and  dust 

•  Reduced  logistics  support. 

The  Army  sponsored  a  successful  four-year  Advanced  Technology 
Engine  (ATE)  competitive  demonstration  program  with  General  Elec¬ 
tric  and  Pratt  and  Whitney  which  substantiated  that  the  perform¬ 
ance  requirements  were  achievable  in  a  full-scale  development 
program.  During  the  latter  period  of  the  ATE  demonstrator  program, 
Army  and  GE  R&M  engineers  conducted  an  in-depth  review  of  the 
current  Army  engine  experience  and  postulated  about  the  future 
operational  and  maintenance  environment,  from  which  the  design 
requirements  were  developed  (Ref.  T700  Case  Study,  p.  IIC-6). 
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Late  1960s  Engine  Experience 
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ENSIP  or  Engine  Structural  Integrity  Program  is  a  formaliza¬ 
tion  of  the  structural  design  and  demonstration  approach  which 
the  Air  Force  has  developed  from  lessons  learned  from  the 
TF30 ,  TF34,  TF39,  TF41,  F100,  F101,  F107,  J85-21,  and  YJ-101 
engine  programs,  as  well  as  others. 

Figure  D-3  categorizes  failure  mechanisms  for  various 
types  of  engine  structures.  Some  structures  exhibit  more  than 
one  failure  mechanism.  For  instance,  turbine  blades  fail  in 
stress  rupture,  high-cycle  fatigue  and  low-cycle  fatigue. 

Since  turbine  blades  operate  with  centr ifugally-induced  stresses 
of  30,000  g's  or  better,  as  well  as  high  thermal  stresses  from 
operating  in  2000-3000°F  gas  temperatures,  while  cooled  with 
1000°F  cooling  air,  and  under  high  aerodynamic  loads  as  well, 
one  can  see  the  challenge  of  the  structural  designs  for  these 
essentially  thin-walled  shells  transitioned  from  relatively 
heavy  platforms. 
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COMPONENT  I YPICA1  MECHANISM  OE  SUMPTION 

FRAMES  CASES  ^  LOW  CYCLE  FATIGUE  «LC«*  ^  CRACKMG  DUE  10  REPETITIVE  APPI  CATION 

OF  CENTRIFUGAL  LOADS  APPLIED  PRESSURES. 
THERMAL  STRESS  AND/OR  FLIGHT  L0A0S 

III AUES  DISKS  ^ 

HEADINGS 

VANES  THIN  0  HIGH  CYCLE  FATIGUE  (HCF)’  ^  CRACKRIG  DUE  TO  HIGH  FREQUENCY  STRESS 

SHELL  STRUCT  OSCILLATIONS  CAUSED  RY  AERODYNAMIC. 

sonic,  dr  mechanical  vibratory 
6XCITATI0H  FORCES 

TUIIRINE  BLADES  +  STRESS  RUPTURE’  0  DEFORMATION  l  CRACKING  DUE  TO  PROLONGED 

Application  of  load  and  temp 

VANES  NOZZLES  ^  OVER  T.EMP  (BURN  THRU)  0  LOCAL  MELTING  DUE  TO  EXCESSIVE  TEMP 

AUGMENT0R 

INFERS 

*  PREEXISTING  AM0/0R  SERVICE  INDUCED  0EFECTS  CAN  ACCELERATE  CRACKING 


FIGURE  D-3.  Engine  Structural  Failure  Modes--Typical 
Structural  Failure  Mechanisms 
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Army  planners  placed  reduction  in  maintenance  and  logistics 
support  requirements  as  the  major  technical  goal,  from  which 
three  primary  design  requirements  evolved: 

•  Design  for  "On-Condition"  Maintenance 

•  Apply  State-of-the-Art  Technology  to  Improve 
Reliability 

•  Make  Maintainability  a  Primary  Design 
Consideration . 

Army  R&M  Engineers  prepared  the  qualitative  and  quantitative 
Reliability  and  Maintainability  requirements  and  objectives  for 
UTTAS  engine  development  for  the  Prime  Item  Development  Specifi¬ 
cation  (PIDS).  These  "R&M"  requirements  represented  the  best 
ideas  collected  from  experienced  specialists  in  Industry  and 
the  Military.  General  Electric  Company  translated  the  require¬ 
ments  to  its  engineering  organization  in  terms  directly  relat- 
able  to  specific,  significant  consideration  that  would  impact 
the  design,  such  as: 

•  A  rigid  set  of  rules  limiting  and  controlling  the 
use  of  lockwire. 

•  All  installation  and  module  replacement  to  be  accom¬ 
plished  with  only  10  of  the  182  hand  tools  in  the 
A07  Army  tool  Box. 

•  The  engine  design  would  not  require  any  special  tools 
at  Aviation  Unit  Maintenance  (AVUM)  or  Aviation  In¬ 
termediate  Maintenance  (AVIM)  levels. 

•  When  oil  level  reading  is  low,  the  oil  tank  will  always 
accept  a  complete  quart  without  detrimental  effect. 

•  The  engine  would  require  no  adjustments  or  trimming 
at  the  field  maintenance  level  for  any  reason. 

•  Mount  locations  would  not  interfere  with  installed 
module  replacement. 

•  No  loose  balance  weights  to  be  exchanged  during  module 
replacement . 

(Ref.  T700  Case  Study,  p.  IIC-8). 
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2.  Design  Alternative  Studies 


Design  trade-off  studies  were  utilized  to  optimize  the  design 
from  the  standpoint  of  meeting  all  of  the  T700  gas  turbine  engine 
requirements.  The  resulting  changes  substantially  improved  the 
R&M  of  the  engine.  Some  of  the  more  significant  trade-off  studies 
were : 


a.  Integral  Inlet  Particle  Separation.  In  the  past,  turbo¬ 
shaft  inlet  separators  were  provided  principally  as  airframe 
parts  of  the  total  installation.  General  Electric's  experience 
with  inlet  separators  in  its  T58  and  T64  engine  installations 
indicated  that  separators  could  be  most  efficiently  designed  as 
part  of  the  engine  (Ref.  T700  Case  study,  p.  IIC-16). 

b.  Top  Mounted  Controls  and  Accessories.  Studies  showed 
that  accessibility  and  ease  of  maintenance  were  significantly 
enhanced  by  top  mounting  the  accessories  and  accessory  drive. 

This  also  provides  protection  from  small  arms  fire  (Ref.  T700 
Case  Study,  p.  IIC-18). 

c.  Axial-Centrifugal  Compressor.  Numerous  design  trade 
studies  were  made  on  the  optimum  compressor  configuration  to  be 
pursued  for  the  next  generation  of  helicopter  engines  in  the 
1500  SHP  class,  which  led  to  the  combination  of  the  axial-cen¬ 
trifugal  design  of  the  T700  compressor. 

Another  trade-off  design  study  was  conducted  on  replaceable 
compressor  blades  in  the  axial  stages  versus  the  'blisk '  design 
in  which  the  blades  are  machined  into  the  wheel.  The  decision 
was  to  go  with  the  "blisk"  construction.  The  T700  axial  compres¬ 
sor  design  evolved  with  only  11  major  parts  (Ref.  T700  Case  study, 
p.  IIC-2G) 
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d.  Combustor.  Conventional  combustors  of  the  1960s  were, 
for  the  most  part,  fabricated  shells  with  duplex  vaporizing  fuel 
nozzles  which  had  poor  durability.  A  more  expensive  machined 
ring  design,  incorporating  central  fuel  injectors  with  better 
durability,  provided  a  much  lower  LCC  than  the  lower  cost  fabri¬ 
cated  design  (Ref.  T700  Case  Study,  p.  IIC-22). 

e.  Gas  Generator  Turbine.  A  more  complicated  turbine 
blade  cooling  scheme  was  rejected  in  favor  of  maintaining  the 
simple  and  more  reliable  radial  convection  system  (Ref.  T700 
Case  Study,  p.  IIC-24). 

f.  Power  Turbine.  Simplification  reduced  the  number  of 
parts  and  material  provided  substitutions  payoffs  in  cost,  R&M, 
life  and  engine  weight  (Ref.  T700  Case  Study,  p.  IIC-24). 

g.  Bearing  and  Lube  System.  A  six-bearing  configuration 
(two  on  the  gas  generator  and  four  on  the  power  turbine)  was 
selected  as  providing  the  optimum  balance  of  rotor  dynamic 
stability,  ease  of  assembly  and  durability  (Ref.  T700  Case  Study, 
p.  IIC-26 ) 

h.  GE12  to  T700  Design  R&M  Requirements  Drove  the  Design 
of  the  T700  Engine.  Early  in  the  program,  the  Army  awarded  a 
supplemental  contract  to  the  T700  contractor  to  perform  a 
Maintainability  Demonstration/Reliability  analysis  on  the  demon¬ 
strator  engine.  This  work  pointed  out  several  areas  where 
maintainability  improvements  were  required  in  the  design. 

During  the  GE12  (ATE)  demonstration  in  early  May  1971,  for 
example,  excessive  times  to  remove  and  replace  the  fuel  controls 
were  experienced  in  addition  to  numerous  hand  tools  and  several 
special  tools  being  required  to  remove  and  replace  the  combustion 
liner.  The  T700  engine  was  completely  redesigned  to  remedy  the 
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identified  Qualitative  and  Quantitative  problems.  A  "module" 
concept  was  adopted  to  allow  replacement  of  entire  subsystems 
witn  a  minimum  of  time  and  mechanical  expertise  without  the 
need  for  special  tools.  The  assembly  and  disassembly  of  modules 
has  been  simplified  for  easier,  quicker  and  more  error-free 
maintenance  (Ref.  T700  Case  Study,  p.  IIC-28). 

3 .  Design  Evaluation  Analysis 

On  the  T700  program,  R&M  analyses  were  required  by  the  RFQ 
and  delineated  in  the  R&M  program  plans.  They  were  performed  as 
an  on-going  process  in  conjunction  with  design.  These  analyses 
resulted  in  several  design  changes  and  were  considered  a  very 
valuable  tool  in  attaining  R&M  objectives  (Ref.  T700  Case  Study, 
p.  IIC-36 ) . 

Computerized  design  evaluation  analyses  were  used  to  track 
and  predict  engine  reliability  and  maintainability  of  the  T700-GE- 
700.  Important  mechanical  and  thermal  stress  analyses  were  also 
used  to  develop  higher  reliability.  These  types  of  evaluation 
analyses  are  also  required  for  ENSIP  Task  II. 

Reference  paragraph  7,  below,  for  CAD  systems  used  in  the  T700 
program. 

4 .  Parts  and  Material  Selection  and  Control 

Parts  and  material  control  was  the  responsibility  of  the 
designer  under  the  cognizance  of  the  reliability  manager  for  the 
T700-GE-700  engine.  This  resulted  in  parts  and  material  control 
which  was  optimized  between  performance  and  reliability.  A  number 
of  special  design  considerations  were  introduced  into  the  General 
Electric  engine  design  to  provide  a  marked  increase  in  engine 
reliability  when  compared  to  previous  engine  designs.  Some  of 
the  more  significant  items  are: 


I 
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a.  Combustor  System 

•  Casing  made  from  INCO  718  for  strength  and  corrosion 
resistance . 

•  Machined  liner,  giving  low  stress  concentration  and 
less  susceptibility  to  cracking. 

•  Constructed  from  Hastelloy  X,  the  T700  combustor 
system  is  designed  to  have  a  minimum  life  of  5000 
hours  when  operated  at  rated  temperature  levels 
at  a  representative  helicopter  loading  schedule. 

Within  this  life,  10,000  start-stop  cycles  are 
allowed  to  account  for  cycling  during  the  mission. 

Thus,  the  total  low-cycle  fatigue  life  is  designed 
to  be  not  less  than  15,000  cycles. 

(Ref.  T700  Case  Study,  p.  IIC-42). 

b.  Turbine  Blades 

•  The  turbine  blades  are  precision  castings  of  R120 
material  with  a  nickel-aluminide  diffusion  coating. 

Rene  120  was  selected  because  it  offers  the  best 
balance  of  capability  in  terms  of  rupture  life  and 
cooling  flow.  R120  is  already  in  development 
production  for  General  Electric's  F101  and  J101 
engine  hardware. 

c .  Nozzles 

•  The  Stage  1  nozzle  is  an  investment  casting  of  X40, 

an  alloy  which  has  a  long  history  of  successful  casting 
Stage  2  nozzles  are  investment  cast  in  segments  of 
two  nozzles  each  in  R80  material.  The  100  percent 
rated  speed  at  SLS,  STD  is  44,720  RPM.  The  rotor 
system  has  been  designed  to  meet  fully  the  overspeed 
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requirement  of  115  percent  of  maximum  rated  Nq  limit, 
plus  ample  margin  for  burst  (15  percent  more). 


d.  Power  Turbine 

e  High  strength  shaft  made  from  INCO  718. 

e  Integral  bucket  tip  shrouds  provide  vibration  damping. 

•  Designed  for  5,000-hr  life. 

(Ref.  T700  Case  Study,  p.  IIC-44). 

e.  Bearings  and  Lubrication 

e  Bearings  are  made  from  M50  material  with  dual  oil  jets 
and  positive  locking  on  inner  and  outer  races. 

e  Size  of  No.  3  bearing  was  increased  to  provide  longer 
life  from  YT  design. 


e  All  main  shaft  bearings  have  oil  squeeze  film  to  dampen 
rotor  vibration  response. 


e  Filter  system  has  3-micron  element  and  impending  bypass 
and  bypass  indicators.  The  capacity  of  filter  was  more 
than  tripled  (.6  square  feet  vs.  1.895  square  feet)  to 
extend  service  life. 

(Ref.  T700  Case  Study,  p.  IIC-44). 


f.  Miscellaneous 

e  Electrical  connectors  are  hermetically  sealed.  Critical 
connectors  have  been  made  "scoop-proof"  by  extending 
the  shells,  which  prevents  bending  of  the  pins  when 
being  mated. 


•  Hydromechanical  unit  has  torque  motor  with  redundant 
windings. 

e  Ignition  exciter  has  redundant  circuitry. 

e  Ignition  system  has  redundant  igniters  and  power  is 
engine-supplied. 

e  System  has  seven  2-element  probes  to  provide  thermo¬ 
couple  redundancy  and  the  immersion  depth  was  changed 
£or  improved  temperature  measurement  accuracy  and 
control. 

(Ref.  T700  Case  Study,  p.  IIC-46) 

5.  Safety  Factors 

The  design  of  the  T700-GE-70Q  engine  and  the  selection  of 
materials  included  safety  factors,  i.e.,  margins  over  rated 
requirements.  Some  of  these  were: 

e  The  frame  struts  are  designed  to  favor  60  percent  power 
condition  with  minimum  chord  to  minimize  losses  during 
off-design  high  swirl  conditions, 
e  Power  turbine  life  is  5000  hours  including  750  hours 
at  100  percent  intermediate  rated  power  and  with  15,000 
cycle  minimum  low-cycle  fatigue  life, 
e  The  turbine  has  a  minimum  overspeed  margin  of  115  per¬ 
cent  of  maximum  rated  speed  limit, 
e  Rugged  torque  shaft  replacing  bell  cranks  and  actuator 
is  integral  with  the  HMU.  Stall  margin  has  been  in¬ 
creased  by  lengthening  the  Stage  1  compressor  blades 
by  0.025  inch  and  by  aerodynamic  redesign  of  the  centri¬ 
fugal  impeller  and  diffuser, 
e  Combustor  designed  for  5,000-hr  life. 
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The  ENSIP's  Task  II  requires  that  stress  concentration  factors 
be  considered  as  a  design  variable.  Figure  D-4  illustrates 
the  effect  on  life  with  the  use  of  different  stress  concentration 
factors.  There  were  no  exceptions  to  safety  factors  in  the  T700. 
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6.  Thermal  Criteria 


Temperature  effects  are  the  major  drivers  of  turbine  blade  and 
vane  life  considered  in  the  design  of  the  combustor,  because  an 
undesirable  temperature  profile  out  of  the  combustor  can  put  maximum 
thermal  stress  at  the  root  area  of  a  turbine  blade,  where  mechanical 
stresses  are  high  due  to  centrifugal,  bending  torsional,  etc. 

The  desired  temperature  profile  places  highest  thermal  stresses 
in  the  area  desired.  The  combustor  structure  itself  must  be 
lightweight,  require  minimum  liner  cooling  and,  although  a  static 
structure,  it  must  withstand  large  and  often  transitory  thermodynamic 
and  mechnical  stresses. 

Modern  turbine  vane  design  has  been  developed  which  alleviates 
the  major  problem  of  leading  edge  cracking  and  hot  gas  ingestion 
with  subsequent  oxidation  and  failure  which  cannot  readily  be 
overcome  by  material  selection  alone.  Positive  AP  through  the 
leading  edge  compartment  provides  out-flow  even  under  cracked 
conditions  and  prevents  oxidation  with  resultant  extended  life. 

The  ability  to  cool  hot  section  parts  with  an  absolute  minimum 
use  of  cooling  air  is  the  challenge  compared  to  those  previously 
discussed  for  the  combustor  (See  Fig.  D-5). 

Figure  D-6  shows  the  sensitivity  to  dwell  effects  of  the 
F100  first  stage  turbine  disk.  The  figure  emphasizes  that  the 
temperature  effects  due  to  dwell  of  as  little  as  30°  to  50°F 
can  result  in  loss  of  1000  cycles  in  disk  rim  life. 
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7.  Computer-Aided  Design 

Computer  programs  were  utilized  in  numerous  ways  in  the 
design  and  R&M  areas  to  assist  the  engineers  in  performing 
analyses  and  design  studies  which  contributed  to  the  success  of 
the  T700-GE-700  engine  in  the  areas  of  reliability  and  main¬ 
tainability.  The  following  are  some  of  the  computer  programs 
which  were  used: 

e  Axial  compressor  airfoil  generation  including  templates 
for  manufacturing 

e  Centrifugal  compressor  aerodynamic  configuration 
e  Turbine  airfoil  generation 
e  Aeromechanical  blade  analyses 
e  Heat  transfer  analysis 
e  Rotor  dynamics 
•  Structures  analyses 

e  Control  system/airframe  rotor  system  dynamic  simulation 
e  Performance  decks  at  various  operational  conditions. 

During  the  course  of  the  Development/Oualif ication  Program, 
many  computer-aided  design  (CAD)  tools  have  been  put  into  place 
at  the  Lynn,  Massachusetts  operation,  and  gradually,  items  such 
as  the  clearance  drawing  for  the  engine  have  been  computerized, 
through  the  advanced  interactive  graphics  system,  so  that  stack-up 
checks  may  be  performed  when  flow  path  changes  are  made  to  show 
the  design  engineer  the  impact  of  such  changes  on  adjacent  engine 
components. 

During  the  development  of  the  final  MQT  design  for  the  power 
turbine.  High  Energy  X-ray  (HEX)  pictures  were  taken  on  an  actual 
operating  T700  development  engine  at  the  General  Electric  outdoors 
testing  facility  at  Peebles,  Ohio.  These  X-rays  were  used  in  con¬ 
junction  with  interactive  graphics  to  establish  the  tip  clearances 
and  shroud  configuration  for  the  MQT  design  power  turbine. 
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In  the  area  of  reliability,  a  computer  math  model  called 
"BRACE"  was  employed  for  assessing/predicting  the  reliability 
Mean-Time-Between-Failure  (MTBF )  for  the  T700  engine  at  any 
point  in  time.  All  failure  data/corrective  action  experience  was 
input  to  this  program  so  that  a  current  MTBF  prediction  was 
available  at  all  times. 

In  a  similar  manner,  a  Maintainability  Math  Model  (M^)  was 
employed  which  showed  the  relationships  between  components,  parts 
and  maintenance  procedures  and  calculated  quantitative  main¬ 
tainability  values. 

At  the  conclusion  of  the  MQT  program,  and  during  the  tran¬ 
sition  from  development  to  production,  a  Producibility  Engineering 
Planning  (PEP)  program  was  put  into  place  to  "productionize"  the 
manufacture  of  the  various  components  for  the  T700  engine.  As  a 
result  of  this  effort,  much  of  the  engine  was  programmed  onto  tape 
for  manufacture  via  numerically-controlled  machines.  This  assured 
a  much  greater  part-to-part  repeatability  and  much  tighter  control 
of  design  tolerances,  resulting  in  better  overall  reliability  (Ref 
T700  Case  Study,  p.  IIC-50). 

The  Air  Force  also  used  computerized  techniques  for  analyses 
and  design  aids  such  as  fatigue  design.  Figure  D-7  shows  a  level 
of  detailed  analysis  around  holes  and  slots  in  a  typical  rim. 

By  careful  analysis  and  design  iteration,  stress  levels  and 
concentrations  can  be  controlled.  For  instance,  after  initial 
design  and  analysis  of  this  part,  it  was  redesigned  to  include 
additional  slots  rather  than  holes,  thus  reducing  the  relative 
local  stress  levels  while  equalizing  the  loading  across  the  part. 
Figure  D-8  depicts  the  ability  to  do  sophisticated  stress 
analysis.  It  is,  of  course,  critically  important  to  know  the 
environment  accurately  as  a  basis  for  this  analysis. 

Figure  D-9  shows  the  extent  of  the  analysis  that  can  now 
be  performed  with  the  capabilities  of  analysis,  instrumentation, 
and  test  techniques  now  available.  The  use  of  eddy  current 


techniques  in  this  case  resulted  m  design  of  several  new  probes 
as  well  as  automated  techniques  for  probe  scanning  and  recording 
This  capability  is  used  throughout  the  industry. 


•  FATIGUE  LIFE  -  VERY  SENSITIVE  TO  DETAIL  DESIGN 
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FIGURE  D-7.  Design  Variables — Fatigue  Design 
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FIGURE  D-8 .  Typical  Disk  Stress  Analysis 


8.  Testability  Analysis 


The  T700-GE-700  engine  required  an  on-condition  monitoring 
and  diagnostic  system.  The  requirements  for  the  design  features 
to  effect  that  capability  were  part  of  the  maintainability 
requirements  given  to  the  designers.  The  maintainability  analysis 
assessed  and  documented  the  fault  isolation  capability  of  the 
on-condition  monitoring. 


9.  Testability  Verification  and  Testin 


The  T700-GE-700  engine  was  designed  for  operation  in  an 
environment  that  is  both  hostile  and  geographically  far  removed 
from  home  base.  Due  to  the  high  reliability  demonstrated  by  the 
T700-GE-700  engine,  only  a  10-hr  inspection  check,  which  is 
accomplished  in  three  minutes,  and  a  periodic  inspection  performed 
at  500  flight-hour  intervals,  which  can  be  performed  on-wing  in 
one  hour,  are  required.  On-condition  monitoring  coupled  with 
simplified  LRU  installation  and  rigging  with  no  required  adjustments 
will  allow  the  T700-GE-700  to  achieve  a  mission  readiness  far 
superior  to  its  predecessors. 


On-condition  operation: 


requires  proven,  reliable,  durable  engine 


utilizes  engine  status  monitors 
— engine  history  recorder 
— torque  reading 
— turbine  temperature 
— oil  level  gauges 
— oil  pressure/temperature 
— filter  impending  bypass  indicators 
— fuel  pressure 


enhanced  by  fault  isolation  features 
— chip  detectors 
— borescope  ports  (7) 

— filter  bypass  indicator 


no  time-scheduled  engine  maintenance  actions. 
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D.  MANUFACTURING 


The  manufacturing  stage  is  as  critical  as  design,  management, 
R&M,  etc.  Manufacturing  can  make  or  break  the  best  of  designs. 

There  is  also  a  need  to  have  manufacturing  engineers  with  equal 
billing  during  the  design  phase.  This  is  especially  true  where 
manufacturing  costs  are  concerned.  There  are  usually  several 
ways  an  item  can  be  manufactured  but  often  the  design  is  driven 
only  by  the  fact  that  one  method  can  deliver  an  item  early  for 
development  with  inadequate  consideration  to  production  costs. 

The  case  studies  pointed  out  activities  related  to  the  engine 
programs  that  have  strong  merit.  This  included  pilot  production 
runs,  early  testing,  corrections  of  problems  found  through  an 
aggressive  failure  analysis/corrective  action  program,  continuous 
review  of  manufacturing  procedures  and  personnel  training,  and 
numerous  program  reviews. 

Failure  reporting  was  the  key  to  ensuring  success.  But 
the  inspection  and  test  criteria  established  to  provide  identi¬ 
fication  of  the  failures  had  to  be  such  that  no  problems  were 
overlooked.  The  studies  indicate  as  much  effort  was  expended 
here  as  on  any  other  part  of  the  program.  Testing  had  to  be 
tailored  and  directions  explicit.  Tolerance  flags  were 
established  that  would  call  attention  to  potential  problem 
areas.  To  this  end,  the  relationship  between  design  and 
reliability  engineers  had  to  be  very  close. 

In-process  inspections  were  also  heavy  players.  Periodic 
checks  and  sample  sizes  must  be  adequate  to  provide  an  absolute 
minimum  of  bad  components  from  entering  the  end  product.  This 
meant  that  explicit  details  had  to  be  developed  and  adhered  to. 

A  last  element  which  is  implied,  but  not  addressed  in 
strong  terms,  is  employee  morale  and  involvement.  A  sense  of 
belonging  is  essential  to  good  quality  control  and  inherent  good 
performance.  If  the  worker  is  not  involved  with  the  program, 
then  the  best  plans  and  designs  will  not  provide  the  system  required. 
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On-condition  maintenance  techniques  are  currently  being 
utilized  successfully  both  in  the  factory  and  in  the  field.  Of 
particular  value  has  been  the  engine  history  recorder  time- 
temperature  integrator  to  measure  hot-part  life  used  and  for 
comparing  the  relative  severity  of  field-test  engine  operation 
with  specification  endurance  test  cycles.  The  engine  chip 
detector  has  proven  to  be  an  effective  means  of  detecting 
incipient  oil-wetted  part  failures.  Borescope  inspection  has 
also  proven  to  be  useful  and  easy  to  do  both  in  the  factory  and 
on  the  wing.  Ground  use  of  the  diagnostic  connector  for  control 
troubleshooting  has  been  effective,  even  though  the  currently 
available  test  box  is  only  a  non-powered  resistance  checker  (Ref. 
T700  Case  Study,  p.  IIC-56). 
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The  T700  case  alludes  to  a  FRACAS  type  system,  but  without 
a  formal  defined  structure.  The  need  during  the  prototype  engine 
period  for  a  "very  early  experience  base"  and  recognition  of  the 
need  for  a  FRACAS  type  system  so  "early  development  problems 
could  expeditiously  be  addressed  and  changes  factored  into  the 
production  engines  in  manufacture"  was  clearly  evident. 

The  T700  design  engineers  and  reliability  engineers  worked 
closely  with  production  engineers  on  a  daily  basis  to  assure  that 
no  problem  went  unnoticed/uncorrected. 

The  T700  design  review  process  is  presented  as  a  form  of 
failure/correction  action  review  board  (Ref.  T700  Case  Study,  p. 

I ID-4 ) . 
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E.  TEST  AND  EVALUATION  (T&E) 


Test  and  Evaluation  is  the  proof  of  the  pudding.  But,  like 
everything  else,  it  must  be  done  correctly.  The  case  studies  of 
the  successful  engine  programs  show  this  in  no  uncertain  terms. 
Requirements  to  which  the  system  has  been  designed  must  be 
verified  and  evaluated  through  all  stages  for  the  acquisition 
cycle.  Planning  and  preparation  require  as  much  care  as  design 
and  control.  Testing  of  a  mechanical  system  requires  that  quali¬ 
fication  tests  be  performed  to  verify  design  limits,  demonstration 
tests  verify  design  operational  requirements,  airframe  integration 
tests  to  verify  generic  flight  and  user  requirements,  R&M  growth 
and  maturity  tests,  and  full  operational  testing  in  the  field. 
Results  from  all  tests  and  assessments  from  field  results  are 
the  follow-on  to  the  formal  tests  to  find  trends  and  design 
errors  that  require  corrective  action  prior  to  full-level  produc¬ 
tion. 

Throughout  the  testing  process,  R&M  must  be  reviewed  con¬ 
stantly.  Growth  of  R&M  in  the  system  must  be  accounted  for  and 
change  incorporated  in  design,  assembly  procedures  or  parts 
requirement  addressed  to  ensure  the  delivered  end  product  meets 
the  required  R&M  levels.  Growth  or  maturation  tests  to  specifi¬ 
cally  address  R&M  in  quantitative  terms  are  required  to  verify 
conclusions  reached  from  data  analysis.  Timing  of  such  tests 
is  vital  in  obtaining  a  true  picture.  Again,  well-defined  require 
ments  are  of  great  importance.  The  case  studies  strongly  point 
to  this  and  stress  the  importance  of  addressing  the  entire  R&M 
area  when  developing  the  test  and  data  review  criteria. 

In  concert  with  the  R&M  tests,  a  factor  of  significance  is 
to  be  able  to  fault  isolate  failures  and  take  corrective  action. 
Such  action  includes  TAAF  actions  as  well  as  simple  mechanical 
adjustments.  The  bottom  line  is  that  the  failure  mechanism 
must  be  known  to  ensure  a  successful  fix.  Failure  reporting 
and  documentation  of  failures  for  trend  analysis  is  important 


to  meet  the  needs  of  the  R&M  staff.  Those  successful  programs 
performed  these  tasks  well  and  as  a  result  met  and,  in  many 
cases,  exceeded  their  required  criteria. 

1 .  Integration  Testing 

Integration  testing  on  the  final  use  platform  is  another 
element  noted  in  the  case  studies  as  "successful  systems."  In 
some  instances,  multiple  platforms  are  designated  as  users  for  a 
system  and  testing  on  each  provides  a  wealth  of  data  pertinent 
to  the  program's  success.  Tailoring  of  such  tests  to  the  schedule 
and  cost  is  a  key  planning  factor.  There  is  also  the  need  to 
make  certain  that  accurate  failure  reporting  is  maintained. 

2.  Design  Limit  Qualification  Tests 

Testing  to  design  limits  addresses  a  very  broad  spectrum 
of  requirements.  This  includes  components  to  environmental 
(i.e.,  vibration,  temperature,  and  contaminant  ingestion)  tests 
developed  in  this  area  and  must  be  well-tailored  to  provide 
data  that  accurately  validate  that  the  requirements  have  been/ 
will  be  met.  In  some  cases,  accelerated  tests  are  required 
and  again  planning  and  well-defined  test  details  must  be  estab¬ 
lished.  Test  severity  levels  must  be  adequate  to  provide  reli¬ 
able  data.  If  the  criteria  are  not  specific  and  broad  inter¬ 
pretation  of  results  is  allowed,  then  credibility  and  success 
likelihood  are  placed  in  jeopardy.  Adherence  to  the  above 
practices  was  stressed  in  each  of  the  case  studies  performed 
for  this  study. 
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The  T700  growth  program  was  incorporated  into  a  Maturity 
Program.  The  overall  program  timing  planned  by  the  Army  provided 
for  competitive  test  (GCT)  of  the  two  different  aircraft,  each  of 
which  was  powered  by  the  same  configuration  T700.  The  test  was 
begun  at  about  the  same  time  that  engine  MQT  was  completed. 

Under  prior  program  standards,  engine  qualification  would  be 
considered  complete  at  this  time  and  the  engine  would  have  been 
committed  to  production.  At  this  point,  a  post-MQT  program  was 
initiated  with  the  goal  of  accumulating  additional  endurance 
experience  and  subjecting  the  engine  to  more  LCF  testing. 

The  overriding  purpose  of  the  Maturity  Program  was  to  provide 
a  mature,  reliable  engine  prior  to  full-rate  production.  To 
accomplish  this,  the  following  objectives  were  established: 
e  Develop  high  initial  mean  time  between  f ailure-require 
overhaul  (MTBFRO) . 

e  Establish  sound  field  maintenance  procedures  and 
intervals. 

e  Identify  unique  installation-related  failure  modes, 
e  Establish  program  for  smooth  transition  to  production 
manufacture. 

The  approach  selected  was  to  conduct  accelerated,  service- 
abusive  tests  so  that  the  required  production  target  dates  were 
ensured. 

A  secondary  benefit  of  the  Maturity  Program  was  that  it 
provided  a  highly  valuable  period  to  resolve  residual  problems 
uncovered  in  the  field  and  factory  programs  and  any  that  might 
evolve  from  the  aircraft  GCT.  In  addition,  a  smooth  transition 
to  production  through  producibility  and  manufacturing  technology 
programs  was  made  possible,  as  well  as  the  implementation  of  cost 
reduction  programs  prior  to  production  (Ref.  T700  Case  Study,  p. 

I IE-32 ) . 
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The  test  program  on  the  T700  did  not  include  a  reliability 
demonstration  test  with  an  accept/reject  criterion  for  reliabil¬ 
ity.  All  development  tests  were  used  to  estimate  the  reliability 
of  the  product.  The  program  did  include  formal  maintainability 
demonstrations. 


a.  Reliability  Demonstration.  From  the  very  beginning  of 
the  T700  development/qualification  test  program,  every  malfunction 
or  discrepancy  was  documented  both  in  the  factory  as  well  as  at 
the  various  UTTAS  and  AAH  Flight  Test  Programs.  Factory  develop¬ 
ment  problems  were  documented  on  development  problems  reports 
(DPRs)  and  field  problems  were  documented  on  DV-7  forms.  Thus, 
every  T700  engine  test  hour,  both  in  the  factory  and  in  the 
field,  represented  an  input  for  the  T700  engine  reliability 
program. 

As  indicated  above,  the  reliability  tracking/analyses  were 
continued  as  part  of  the  post-MOT  Maturity  and  Life  verification 
Testing. 

Approximately  35,000  engine  operating  hours  were  logged  by 
the  time  the  first  T700-GE-700  production  engine  was  delivered  in 
March,  1978,  thus  providing  an  excellent  reliability  demonstra¬ 
tion  base  upon  which  to  base  the  engine's  mean  time  between 
failures  (MTBF ) . 

With  the  introduction  of  T700-powered  Blackhawks  into  the 
field  in  1979,  a  field  tracking  system  was  initiated  (Ref.  T700 
Case  Study,  p.  IIE-46). 


b.  Maintainability  Demonstration.  As  indicated  in  the  T700 
case  study,  a  very  early  maintainability  demonstration  was 
funded  by  the  U.S.  Army  and  was  accomplished  on  the  ATE  (GE12) 
demonstrator  engine  in  1971.  With  the  award  of  the  development 
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contract  in  March,  1972,  the  Maintainability  Program  Plan  for 
the  T700  engine  was  set  into  action.  The  following  paragraph 
was  extracted  from  this  plan.  The  total  plan  describes  the 
three  official  maintainability  demonstrations  required  by 
contract : 

Maintainability  Checks/Demonstrations  -  The  contractor 
shall  conduct  the  following  maintainability  checks  and 
demonstrations  and  shall  coordinate  the  effort  with  all 
other  interfacing  specialty  disciplines  and  the  ILS 
program.  The  plans  for  conducting  the  first  engine  tear- 
down  and  the  maintenance  evaluation  will  be  submitted  in 
the  monthly  progress  report  in  accordance  with  data  item 
DI-R-1741  and  addendum  dated  19  March,  1971,  part, 
component  or  subsystem  test  plan(s). 

(Ref.  T700Case  Study,  p.  IIE-52). 
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The  purpose  of  Operational  Testing  is  to  place  the  system  in 
a  full-up  configuration  and  verify  its  success  in  meeting  users 
needs  and  requirements.  Hands-on  experience  is  gained  and  faults 
are  identified  by  the  user  that  the  designer  may  not  have  fully 
anticipated.  Adequate  and  accurate  fault  reporting  during  this 
phase  is  essential  to  providing  a  true  picture  of  the  program's 
compliance  to  the  requirements  and  users  needs.  it  is  here  that 
the  first  true  evaluations  of  R&M  factors  and  requirements  are 
seen.  The  feasibility  of  the  maintenance  concept  is  assessed  and 
attainment  of  requirements  is  checked;  questions  such  as 
accessibility  of  components,  use  of  captive  and  foolproof 
fasteners  and  connectors,  etc.,  are  truly  addressed.  The  results 
allow  the  full  production  effort  to  begin  and  start  the  data 
collecting  for  "in-service  assessments"  to  begin. 

A  helicopter  development  "first"  was  pioneered  by  a  joint 
U.S.  Army/GE  team  by  simultaneously  developing  a  new  turboshaft 
engine  and  four  different  experimental  helicopters.  The  four 
aircraft  were  involved  in  two  major  flyoff  competitions,  UTTAS 
and  AAH .  The  T700  and  UTTAS  were  developed  simultaneously  under 
aspices  of  the  Materiel  Development  and  Readiness  Command  (DARCOM) 
UTTAS  Program  Manager's  office,  with  AAH  starting  almost  12 
months  later  (See  Fig.  D-10). 

All  four  types  of  twin-engined  experimental  helicopters 
used  identical  versions  of  the  YT700-GE-700  turboshaft  engine — 
unprecedented  for  simultaneous  engine/helicopter  development  (Ref. 
T700  Case  Study,  p.  IIE-68). 

A  key  factor  in  successfully  integrating  a  single-engine 
configuration  into  four  helicopters  is  a  thorough,  pre-field- 
test  propulsion  system  integration  effort:  factory  engine  envi¬ 
ronmental  and  "fleet  leader"  testing,  repeated  Army/GE/AVM 
design  and  test  reviews,  and  factory  performance  and  vibrational 
testing  for  each  installation. 
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Engine  reliability  data  and  installation  "lessons  learned" 
from  the  18,000  engine  test  hours  accumulated  during  the  UTTAS 
and  AAH  Programs  represent  an  invaluable  opportunity  to  examine 
four  different  propulsion  systems  in  a  concentrated  time  period 
and  apply  the  "lessons  learned"  from  actual  operational  testing 
into  early  corrective  actions  resulting  in  accelerating  the 
reliability  growth  of  the  engine  (Ref.  T700  Case  Study,  p.  IIE-68) 

The  Army's  UTTAS  development  philosophy  included  testing 
that  went  far  beyond  the  normal  extent  of  the  typical  competitive 
flyoff  program.  Both  Boeing  Vertol  and  Sikorsky  were  required  to 
demonstrate  all  production  aircraft  systems  in  the  working  Army 
environment  prior  to  production  contract  award.  Both  AVM's  had 
to  produce  a  brand  new  aircraft  and  make  it  perform  as  advertised, 
but  they  also  had  to  live  under  the  working  Army's  microscopic 
evaluation.  Naturally,  that  meant  taking  prototype  aircraft  into 
the  mud  of  Ft.  Campbell  and  meeting  predetermined  reliability 
and  maintainability  goals  using  the  standard  Army  field  team. 

The  Army  was  also  looking  for  a  full  exploration  of  the 
UTTAS  flight  envelope,  demonstration  of  C-141  and  C-5A  air  trans¬ 
portability,  and  the  ability  to  perform  defined  UTTAS  missions 
in  all  types  of  adverse  environmental  conditions — icing,  heat, 
cold,  night  flying,  and  forward  operating  sites.  Under  DoD  fly- 
before-buy  concepts,  every  effort  was  made  to  make  sure  that  the 
U.S.  Army  knew  what  they  were  buying  and  that  the  system  could 
successfully  live  with  the  field  Army  (Ref.  T700  Case  Study,  p. 
IIE-70 ) . 

Government  Competitive  Tests  (GCT)  started  in  March  1976 
at  Ft.  Rucker,  AL,  using  two  aircraft  from  each  AVM.  In  May, 

1976,  aircraft  performance  and  handling  quality  testing  began  at 
Edwards  Air  Force  Base  with  the  third  Army-owned  aircraft  per 
AVM.  During  the  next  seven  months,  these  six  aircraft  achieved  a 
total  of  3800  engine  test  hours  at  six  different  operating  test 
sites:  Army  User  Evaluations  were  conducted  at  Ft.  Rucker,  AL, 
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Ft.  Campbell,  KY,  and  at  a  high-altitude  Bishop,  CA,  site. 

Aircraft  icing  evaluation  was  conducted  at  Ft.  Wainwright,  Alaska. 
Each  AVM's  GTV  was  also  used  for  cold/hot  environmental  ground 
testing  at  Eglin  Air  Force  Base,  FL. 

Primary  purpose  of  GCT  testing  was  to  put  the  production 
"prototype"  UTTAS  under  a  rigid  Army  User  evaluation  with  the 
main  emphasis  on  operational  realism.  Not  only  were  user  flight 
tests  flown  by  randomly  selected  Army  pilots,  but  all  AVUM  level 
(flight  line)  maintenance  was  performed  by  representative  Army 
mechanics,  all  of  whom  were  monitored  by  an  "Army"  of  reliability 
and  maintainability  data  collectors. 

Almost  50  percent  of  total  UTTAS  experience  occurred  with  Army 
pilots  and  mechanics  operating  in  the  User's  world — a  resounding 
affirmation  of  the  fly-before-buy  concept. 

During  the  entire  BED  Phase  and  GCT,  GE  technical  represen¬ 
tatives  documented  every  engine  problem/discrepancy  and  these  data 
were  factored  into  the  Reliability  Analyses/Predictions  and  also 
provided  a  'real  world'  operational  experience  base  that  was  un¬ 
precedented  on  other  engine  development  programs.  This  phase  of 
the  program  cannot  be  overemphasized  since  it  provided  a  direct 
feedback  to  the  contractor  which  expedited  corrective  actions  (Ref. 
T700  Case  Study,  p.  IIE-76). 

a.  Maturity  Program 

Following  the  completion  of  the  Government  Competitive  Tests 
(GCTs ) ,  the  Sikorsky-built  YUH-60  was  selected  for  production 
and  designated  as  the  Blackhawk.  A  follow-on  Maturity  Flight 
Test  Program  contract  was  awarded  to  both  Sikorsky  and  General 
Electric.  The  three  prototype  YUH-60 's  and  the  GTV  were  subjected 
to  a  limited  update  program  and  the  YT700  engines  were  returned 
to  the  factory  and  updated  to  incorporate  several  fixes  which  had 
been  identified  during  both  the  Development/Qualif ication  Program 
and  the  BED  Phase  and  GCT  field  programs.  These  engines  were 
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Reliability  and  Maintainability  monitoring  by  Army  data 
collectors. 

The  AAH  Program  was  originally  scheduled  to  be  16  months 
shorter  than  the  UTTAS  competitive  cycle — 35  months  from  initial 
contract  awarded  to  completion  of  the  GCT  Program.  Although  cost 
increases  and  aerospace  material  shortages  in  the  late  1973,  early 
1974  time-period  contributed  to  a  six-month  program  slippage,  the 
AAH  Phase  I  Program  and  GCT  were  still  completed  12  months  faster 
than  the  UTTAS  Program.  Still,  the  AAH  flight-test  program  aver¬ 
aged  20  percent  more  YT700  engine  operating  hours/aircraft/month 
than  the  UTTAS  Army  program  [63  vs.  53  hours]  (Ref.  T700  Case 
Study,  p.  IIE-80 ) . 

Phase  I  testing  (equivalent  to  the  UTTAS  BED  Phase)  started 
with  GTV  operation  in  June  1975.  By  the  end  of  ground  testing  in 
May  1976,  the  two  competitors  had  completed  1300  hours  of  XT  and 
YT700  engine  operation.  Unlike  UTTAS,  which  utilized  GTV  testing 
to  demonstrate  long-term  "fleet  leader"  reliability,  AAH  GTV 
testing  was  limited  to  50  hours  of  preflight  aircraft  qualifica¬ 
tion  and  subsequent  follow-on  qualification  (up  to  150  hours)  of 
the  propulsion  system  and  drive  train.  AAH  flight  testing  was 
initiated  in  September  1975  and,  by  the  conclusion  of  Phase  I  in 
May,  1976,  both  Hughes  and  Bell  had  accumulated  1700  YT700  engine 
operating  hours  with  the  two  flyable  aircraft  operated  by  each 
company.  Since  UTTAS  testing  had  already  been  in  progress  for 
12  months,  many  initial  engine  operating  problems  and  trouble¬ 
shooting  procedures  had  already  been  resolved,  thus  significantly 
speeding  engine/airframe  integration  during  the  initial  portion 
of  AAH  flight  test.  The  AAH  flight  test  program  was  supported 
by  28  YT  engines  evenly  distributed  between  the  two  test  sites 
and  four  SRD  engines  kept  at  GE's  Lynn,  MA  facility  for  rapid 
verification  and  qualification  of  field-related  problems. 
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The  primary  goal  of  Phase  I  flight  testing  was  a  preliminary 
exploration  of  the  aircraft  performance  envelope  and  validation 
of  basic  handling  qualities.  In  addition,  some  limited  propulsion 
system  flight  surveys  were  conducted  by  joint  GE/AVM  teams  and 
the  basic  2.75-inch  rocket  and  30-mm  gun  systems  were  demonstrated. 
All  ground  and  flight  testing  was  conducted  at  each  AVM's  flight 
test  facility:  Hughes  helicopters  at  Palomar,  CA  and  Bell  Helicopter 
at  Arlington,  TX  (Ref.  T700  Case  Study,  p.  IIE-82). 

The  Phase  2  program  was  a  56-month  full-scale  engineering 
development  wherein  the  two  Hughes  helicopters  from  Phase  1  were 
modified  to  the  latest  configuration.  Three  more  helicopters  were 
built,  and  development  of  the  HELLFIRE  missile,  30-mm  cannon,  and 
27.5-mm  rocket  subsystems  completed.  Also  competitively  developed 
were  the  target  acquisition-designation  sight  and  pilot's  night- 
vision  sensor.  The  subsystem  and  all  mission  equipment  have 
been  integrated  and  thoroughly  tested.  The  Army  operational  Test 
II  was  completed  with  the  APACHE  accumulating  over  400  flight 
hours  during  June-August  1981.  A  total  of  over  4,000  test  hours 
were  flown  on  the  YAH-64  prototypes.  The  residual  AAH  weapons 
system  testing  and  other  related  essential  activities  were  also 
completed. 

Long-lead- time  contracts  for  production  of  the  APACHE  were 
awarded  in  February  1981.  The  initial  production  contract  was 
awarded  15  April  1983. 

During  the  entire  AAH  Flight  Test  Program,  GE  technical  repre¬ 
sentatives  have  documented  all  engine  discrepancies  and  these  data 
have  been  processed  through  the  Lynn  Product  Data  Center  and  T700 
Reliability  Operation  for  inclusion  in  the  Reliability  Analyses/ 
Predictions. 
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The  in-service  assessments  are  important  to  the  acquisition 
program  and  R&M  growth  because  of  the  identification  of  problem 
areas  and  trends  not  seen  earlier.  These  assessments  also  provide 
the  supply  and  maintenance  information  needed  to  refine  the 
logistics  support  requirements  and  identify  critical  areas  where 
actions  are  required.  It  is  important  that  in-service  assessments 
be  started  as  early  as  possible  in  the  program.  In  the  T700  pro¬ 
gram  they  were  started  prior  to  the  Government  Competitive  Tests. 
By  now  the  T700  has  completed  one-quarter  of  a  million  operating 
hours  and  is  receiving  exceptionally  high  marks. 

Using  a  90-day  rolling  average  the  shop  visit  rate  for  all 
causes  during  1982  hovered  around  0.5  which  is  the  maturity  goal 
for  the  engine.  Engine-caused  removals  account  for  about  half 
of  the  rate,  or  approximately  0.25,  which  is  comparable  to  large 
commercial  engines.  That  converts  to  an  engine-caused  MTBR  of 
approximately  4,000  hours — quite  a  record  when  compared  to  1960's 
engine  history.  As  a  comparison,  the  T700  shop  visit  rates  for 
various  causes  generally  are  running  from  zero  to  50  percent  that 
of  the  earlier  powerplants  (see  Figure  D-ll).  It  may  also  be 
noted  that  earlier  engines  had  a  TBO  while  the  T700  does  not 
because  it  has  been  "on-condition"  from  day  one. 

A  maturity  goal  of  1.6  removals  per  1,000  engine  flight 
hours  for  all  causes,  for  LRUs,  was  also  established.  In  the 
beginning  of  1982,  LRU  removals  were  running  approximately  2.0. 
That  rate  has  dropped  sharply  and  is  now  close  to  the  maturity 
goal.  Incorrect  LRU  removals  have  been  in  the  range  of  30  to 
50  percent;  particularly,  the  HMU  and  ECU.  It  has  been  found  that 
the  ease  of  removal  and  replacement  can  lead  to  "gang"  trouble¬ 
shooting  rather  than  a  logical  fault-free  analysis  laid  out  for 
the  mechanic  in  the  engine  field  technical  manual.  Through  im¬ 
proved  troubleshooting  reviews  by  GE's  product  support  organiza¬ 
tion  and  assistance  by  on-site  technical  representatives,  the 
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incorrect  removal  rate  is  decreasing.  In  addition,  GE  has 
developed  a  system  analyzer,  now  being  evaluated  by  the  Army, 
which  is  designed  to  isolate  those  sometimes  difficult  electrical 
circuit  faults.  The  Army  is  currently  reviewing  the  very  success¬ 
ful  field  history  of  two  prototypes  to  determine  if  they  will 
award  a  limited  production  contract. 

Flight  line  maintenance  represents  60  percent  of  the  mainte¬ 
nance  actions.  A  comparison  between  the  T700  and  lSSO's  engines 
of  the  man-minutes  required  to  remove  and  replace  four  elements  is 
shown  in  Fig.  D-12. 

The  maintainability  requirements  of  the  T700  are  exhibited 
most  dramatically,  perhaps,  by  the  fact  that  approximately  only 
one  man-year  of  corrective  maintenance  man-hours  has  been  required 
on  the  T700  during  its  first  quarter  million  hours  of  service. 

This  includes  all  T700  engines  at  all  Blackhawk  operating  sites. 
There  have  even  been  complaints  from  maintenance  officers  about 
maintenance  personnel  losing  proficiency  because  there  is  not 
enough  maintenance  activity  on  the  T700  to  keep  them  sharp.  This 
is  perhaps  true  in  spite  of  the  fact  that  2  to  3  fewer  engine 
mechanics  have  been  assigned  to  Blackhawk  units  than  to  "Huey" 
companies  (Ref.  T700  Case  Study,  p.  IIE-108). 

Dedicated  on-site  Field  Monitors  collected  and  edited  R&M 
data  on  the  UH-60A  Blackhawk  program  as  part  of  the  Sample  Data 
Collection  (SDC)  program.  Additional  Blackhawk  R&M  data  were 
collected  from  the  three-year  T700  Reliability  Improvement  Pro¬ 
gram  and  from  CRIM  (Comprehensive  Record  for  Intensive  Management) 
records  which  were  completed  each  time  the  user  removed  one  of 
the  components  selected  for  intensive  management." 

F.  OBSERVATIONS 

In  summary,  the  T700  at  250,000  hours  has  established  an 
outstanding  record  compared  to  mature  prior  generation  engines. 

91/13-10 


D-53 


1 


One  result  of  the  improved  R&M  characteristics  has  beta  reduction 
in  engine  spares  to  15  percent  rather  than  50  percent,  a  saving 
of  $400  million.  Major  conclusions  to  be  drawn  from  the  develop¬ 
ment  experience  with  this  engine  are: 

•  The  Army's  careful  preparation  of  the  reliability 
and  maintainability  requirements,  and  its  insist¬ 
ence  that  they  be  met,  are  already  producing  a  pay¬ 
off  in  low  shop-visit  rates  and  line  replaceable 
unit  removal  rates.  GE  and  the  Army  now  can 
concentrate  more  on  improvements  to  the  engine 
rather  than  "fix-it"  programs  (Ref.  T700  Case  Study, 
p  IIE-16) 

•  The  6.3A  demonstrator  engine  program  extended 
over  an  11-year  period  without  funding  disrup¬ 
tions  which  allowed  an  orderly  development 
without  fluctuations  in  personnel.  Approximately 
$200  million  in  RDT&E  funds  were  provided  for  this 
effort. 

•  The  early-on  warranty  allowed  the  contractor  to 
have  configuration  control  and  total  logistics 
latitude  to  introduce  corrective  actions  quickly 
before  the  fleet  had  a  chance  to  grow,  which  would 
make  corrective  action  more  costly  and  complex  to 
implement  (Ref.  T700  Case  Study,  p.  IIE-116). 

As  a  result  the  T700  has  already  developed  a  reputation  for 
being  a  low  cost  of  ownership  engine  that  is  highly  reliable. 

In  retrospect,  it  is  also  possible  to  see  ways  in  which  the 
development  of  engines  could  be  done  better.  One  major  recommenda¬ 
tion  that  originated  with  the  contractor  is  that  Accelerated  Mis¬ 
sion  Testing  (AMT)  be  started  much  earlier  in  the  program.  The 
reasons  for  this  recommendation  are  as  follows: 

Looking  at  the  T700  test  programs  in  chronological  order, 
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it  is  easy  to  see  the  learning  process  develop  as  the  factory 
test  program  becomes  more  and  more  oriented  toward  Accelerated 
Mission  Testing  (AMT)  type  of  tests.  The  T700  Qualification  Pro¬ 
gram  included  two  150-hour  MQT  tests  as  well  as  two  1,750-cycle 
LCF  tests  to  gain  confidence  in,  and  to  demonstrate  the  integrity 
of,  components  directly  affected  by  LCF  damage.  More  AMT-type 
testing  could  have  accomplished  the  same  results  using  fewer 
engines  and  less  time.  The  potential  cost  savings  are  evident. 

To  provide  additional  confidence  in  the  long-term  integrity 
of  engine  components,  1,000-hour  MQT  endurance  tests  were  initi¬ 
ated  in  1976.  The  first  real  Accelerated  Mission  Testing  was 
initiated  with  the  1,500-hour  Blackhawk  ASMET  test.  It  was  not 
until  later,  in  1978,  however,  under  the  Component  Improvement 
Program  (CIP),  that  substantial  AMT  testing  commenced  in  which 
engine  parts  were  exercised  in  ELCF ,  EFTC  and  ETAMP  in  proportions 
representative  of  anticipated  field  usage. 

For  many  years  now,  the  MQT- type  test  cycle  has  served  as 
the  standard  used  to  qualify  alternate  sources.  It  is  suggested 
here  that  the  time  has  come  for  a  new  standard  to  be  established — 
the  AMT  test  cycle.  This  focus  on  AMT  testing  should  be  integrated 
into  the  Full-Scale  Engineering  Development  Program  at  the  outset 
of  FSD.  The  objective  is  to  achieve  early  maturity  for  the 
engine  design  during  the  FSD  program  and  to  eliminate  or  mini¬ 
mize  the  need  for  a  follow-on  Maturity  Program  Phase.  The  T700 
development  program  incorporated  the  Maturity  Program  concept  con¬ 
ceived  by  the  U.S.  Army  to  identify  and  fix  component  deficiencies 
which  may  occur  in  engines  that  are  not  fully  mature  before 
entering  production.  This  program  has  significantly  contributed 
to  the  T700's  successful  field  experience. 

It  is  recommended  that  future  Full-Scale  Engineering 
Development  Programs  focus  on  AMT  testing  from  the  outset  of  the 
program.  AMT  testing  accomplishes  much  more  in  establishing 
parts  life  integrity  and  exercises  all  parts  in  production  to 
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their  intended  field  usage.  AMT  testing,  as  part  of  an  overall 
engine  life  management  program  coupled  with  a  fleet  leader  pro¬ 
gram  and  sufficient  engine  usage  monitors  will  lead  to  even 
higher  standards  of  engine  reliability  and  lower  life-cycle  costs 
than  has  been  achieved  in  earlier  programs  (Ref.  T700  Case  Study, 
p.  IIA-58 ) . 

In  addition,  based  on  the  T700  experience,  some  corrective 
actions  have  been  identified  by  the  Army  for  current  developments 
as  follows: 

•  Increase  test  hours  in  the  6.3A  program  to  allow  for 
limited  LCF  and  abusive  testing.  This  action  has 
already  been  initiated  on  the  800  HP  ATDE  and  the 
5000  HP  MTDE  programs. 

•  Validate  Inlet  Particle  Separator  (IPS)  performance 
as  part  of  6.3A  program.  This  has  been  incorporated 
into  the  ATDE  and  MTDE  program. 

•  Establish  common  power  turbine  speed  for  competitive 
6.3A  demonstrator  programs.  This  has  been  incorpora¬ 
ted  into  the  ATDE  and  MTDE. 

•  include  producibility  and  design  to  unit  production 
cost  from  the  outset.  These  items  have  been  incor¬ 
porated  into  the  ATDE  and  MTDE. 

•  Refine  the  balance  between  reliability /maintainability . 

The  current  MTDE  reflects  a  better  balance  (e.g.,  reduced 
number  of  modules  for  increased  reliability). 

•  Eliminate  the  time  gap  between  advanced  development 
and  engineering  development  programs.  This  requires 
a  management  decision  by  DoD  and  DA  to  commit  to 
development  of  generic  engines  prior  to  the  firm  air¬ 
craft  requirement. 
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There  are  also  specific  problems  which  continue  to  appear 
as  more  extensive  field  experience  is  accrued.  A  review  of  the 
conditions  of  field  returned  T700  engines  by  the  AVRADCOM  Direct¬ 
orate  of  Product  Assurance  led  to  the  following  observations: 

"Our  observations  were  that  most  of  the  returns  were  the 
result  of  externally-induced  failures  as  a  result  of  the  aircraft 
power  extraction  shaft  failures.  The  areas  of  high-cost  repairs 
were  FOD  damage  to  the  compressor  and  diffuser  section.  The 
second  observation  was  on  engines  returned  for  low  power  had 
eroded  and  contaminated  (dirty)  compressor  sections.  GE  was  run¬ 
ning  engines  as  received  to  obtain  baseline  data  performance  deteri¬ 
oration.  Engines  with  low  power  have  the  potential  for  extended 
service  if  adequate  compressor  cleaning  is  accomplished,  as  demon¬ 
strated  by  the  GE  test  and  subsequent  cleaning.  Reducing  the 
compressor  efficiency  results  in  the  loss  of  efficient  power,  which 
increases  both  direct  and  indirect  O&S  costs.  A  specific  recom¬ 
mendation  is  made  for  all  future  propulsion  systems  that  a  water 
wash  system  with  tanks  be  made  a  component  of  any  modern  technol¬ 
ogy  engine.  The  purpose  is  to  realistically  facilitate  a  method 
to  maintain  compressor  efficiency;  a  water  wash  system  must  be 
part  of  the  pilot's  operating  procedure.  Specifying  it  as  a 
maintenance  function  (i.e.,  every  100  hours  or  as  required),  is 
labor  intensive  and  not  complied  with,  per  our  observation.  The 
third  observation  is  a  general  RAM  improvement  program  of  gas 
turbines.  Recommend  that  a  Product  Assurance  life-cycle  RAM 
plan  be  adopted  to  ensure  that  the  contractor  and  Government 
have  within  their  RAM  management  concept  a  commitment  to  provide 
life-cycle  feedback  subsequent  to  production  and  fielding.  The 
prime  manufacturer's  RAM  concepts  should  include  a  Government 
commitment  to  allow  the  contractor  to  have  a  RAM  evaluation  pro¬ 
gram  via  a  life-cycle  minimum  repair  allocation  of  engines. 

The  purpose  is  to  ensure  that  feedback  is  maintained  to  evaluate 
production  and  overhaul  changes  introduced  during  the  life  cycle 
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that  can  be  correlated  back  to  the  original  production  margins 
and  performance  factors.  The  current  practice  of  internal 
military  overhauls  precludes  a  life-cycle  product  assurance 
feedback  system  to  the  prime  and  may  allow  him  to  repeat  design 
and  manufacturing  processes  which  are  detrimental  to  RAM  and 
life-cycle  costs  because  of  the  lack  of  a  quantitative  feedback 
system  under  his  control. 

It  is  the  general  observation  that  the  Army  has  made  a  signi¬ 
ficant  RAM  contribution  in  supporting  the  T700  as  a  highly 
reliable  propulsion  system  and  should  continue  the  same  type  of 
approach  on  a  new  generation  propulsion  system;  and,  when  produc¬ 
tion  requirements  are  adequate  two  competitive  contractors  should 
be  maintained  in  order  for  the  Government  to  obtain  the  benefits 
of  competition." 

Extensive  field  experience  has  also  surfaced  some  potential 
design  improvements  as  follows; 

•  Integral  Inlet  Particle  Separator.  In  spite  of  the  85 
percent  efficiency  of  this  device  in  separating  sand  and 
dust,  it  remains  inadequate  to  the  engine  requirements. 
Larger  particles  peening  the  Stage  1  blades  are  a  signifi¬ 
cant  factor  in  limiting  on-wing  life  due  to  performance 
loss.  In  moderate  environments  (i.e..  Ft.  Campbell/Ft. 
Bragg)  engine  performance  can  degrade  to  the  reject 
limit  in  1000  to  1400  hours  of  operation. 

•  Axi-Centr ifugal  Compressor.  The  "blisk"  concept 
appears  to  have  been  sound,  with  a  single,  not¬ 
able  exception.  The  first  stage  "blisk"  incurs  a 
large  amount  of  FOD,  and  degrades  rapidly  due  to 
sand  peening  of  the  blade  leading  edges.  In 
retrospect,  a  design  allowing  for  field  replaceable 
Stage  1  blades  would  have  offered  superior  main- 
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•  Anti-Ice  Start  Bleed  Valve,  This  dual  function 
valve  is  now  in  its  second  redesign.  It  appears 
that  in  an  effort  to  reduce  the  number  of  systems/ 
parts,  an  overly  complex  and  very  sensitive  device 
resulted.  A  large  number  of  malfunctions  of  this 
device  have  resulted  in  mission  restriction  in 
some  theaters  and  excessive  downtime.  In  retro¬ 
spect,  separate  anti-ice  and  start/acceleration  bleed 
systems  would  have  been  desirable. 

•  Electronic  Control  Unit.  This  unit  remains  very 
costly,  and  labor  intensive  to  manufacture,  repair 
or  modify.  Electronics  state-of-the-art  has 
advanced  far  beyond  this  design,  leaving  it  obsoles¬ 
cent.  Due  to  cost  and  support  limitations,  intensive 
logistics  management  is  required  to  provide  adequate 
fleet  support. 
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APPENDIX  E 


DEVELOPMENT  OF  DIAGNOSTICS 


The  development  of  diagnostics  is  an  immature  discipline  when 
compared  to  reliability.  The  terms  for  contractual  and  operational 
reliability  are  not  the  same,  but  they  are  predictable  and  can  be 
related.  In  diagnostics,  there  is  no  clear  definition  of  require¬ 
ments  which  can  be  used  for  contracting,  is  understandable  to  a 
designer,  and  is  closely  related  to  a  field  measure.  For  reli¬ 
ability,  there  are  well-understood  design  tools  for  analysis  of 
the  stresses  which  cause  failures  as  well  as  predicting  failure 
rates  for  components,  subsystems  and  systems.  Design  tools  for 
diagnostics  are  much  less  structured  and  practiced. 

In  reliability  testing,  there  are  proven  techniques  for 
simulating  the  operational  stresses  an  equipment  will  undergo, 
weeding  out  the  causes  of  unreliability  and  verifying  the  potential 
system  reliability.  Diagnostics  testing  techniques  are  much  less 
mature.  Though  fault  insertions  are  performed,  they  are  not 
good  predictors  of  field  performance.  They  will  indicate  whether 
there  are  problems,  but  success  in  such  a  demonstration  i s  no 
guarantee  of  a  good  design.  Thus,  demonstration  by  fault  inser¬ 
tions  are  necessary,  but  not  sufficient. 

During  early  operational  introduction,  the  assessment  of 
reliability  is  much  more  straightforward  than  diagnostics.  There 
are  some  problems  with  using  field  data  to  assess  reliability 
during  full-scale  field  operations.  However,  the  problems  with 
the  data  systems  are  understood  well  enough  that  field  reliability 
data  are  useful  for  some  management  purposes  (trending  if  not  in 
an  absolute  sense).  The  data  system  is  almost  useless  for  diagnos¬ 
tics.  It  does  not  reflect  the  method  of  detection/isolation, 
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huiaan  intervention  in  the  decision  process,  or  troubleshooti ny 
time  factors.  This  lack  of  knowledge  in  the  diagnostics  arena 
(contracting,  requirements,  design,  testing,  employment)  presents 
a  significant  challenge  to  the  military  and  industry  communities 
in  the  improvement  of  current  systems  and  the  acquisition  of 
future  weapon  systems.  Fundamental  work  is  required  in  all 
these  facets  of  weapon  program  development  to  produce  acceptable 
diagnostic  capabilities  in  the  field. 


A.  DIAGNOSTICS  TERMINOLOGY 


A  fundamental  indication  of  a  problem  in  diagnostics  is  the 
many  inconsistent  and  overlapping  terms  and  definitions  used  to 
describe  and  measure  diagnostics  capability.  One  of  the  recom¬ 
mendations  of  the  report  is  to  generate  and  standardize  a  compre¬ 
hensive  set  of  diagnostics  terms.  This  section  does  not  do  that; 
it  is  simply  a  discussion  of  the  terms  in  the  report  so  that  the 
reader  can  relate  his  own  experience  with  diagnostics  terms  to 
those  presented  here. 

Describing  Capability;  From  the  perspective  of  the  prime 
system  user,  the  basic  functions  of  a  built-in-test  capability 
can  be  categorized  into  four  groups:  system  health  monitoring, 
expanded/initiated  fault  detection,  maintenance  health  monitoring/ 
fault  duplication/correction  verification,  and  maintenance  fault 
isolation  (on-line  or  off-line).  The  terminology  used  to  describe 
these  functions  may  not  distinguish  between  them  and  the  terms 
may  be  used  differently  from  program  to  program.  The  important 
point  is  that,  without  standard  terms,  a  lengthy  communication/ 
definition  process  is  required  for  all  parties  (materiel  developer, 
contractor,  user,  and  tester)  to  understand  the  objectives  and 
capabilities  of  diagnostics  in  a  given  program. 

In  general,  most  automated  diagnostic  systems  have  two 
basic  capabilities:  health  monitoring/fault  detection  and  fault 
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isolation.  The  first  capability  usually  runs  whenever  the  system 
is  turned  on,  it  monitors  functions  without  interrupting  system 
operation,  and  it  requires  no  initiation  from  the  operator.  In 
some  programs  (e.g.,  F-16),  this  is  called  self-test  or  continuous 
BIT  which  emphasizes  the  non-i nterrupti ve  characteristic  from  the 
crew’s  perspective.  In  other  programs  (F/A-18),  this  capability 
is  called  periodic  BIT  which  emphasizes  that  the  monitoring 
functions  are  performed  at  specific  times  during  system  operation. 
Most  systems  also  incorporate  a  capability  for  the  operator  to 
initiate  further  test  activity  in  order  to  determine  the  extent 
of  the  problem.  This  usually  requires  some  interruption  of  system 
operation  and  is  called  BIT  in  the  F-16,  and  initiated  BIT  in  the 
F-l 5  and  F/A-18. 

Both  the  self-test  and  the  interruptive  or  initiated  capability 
can  also  be  used  as  the  maintenance  fault  detection  tool  to 
detect  malfunctions,  confirm  reported  malfunctions,  and  verify 
actions  taken  to  correct  malfunctions.  The  initiated  BIT  may 
also  be  designed  as  the  maintenance-fault-isolation  tool.  However, 
if  off-line  diagnostic  routines  are  loaded  to  perform  this  task, 
the  capability  is  sometimes  given  the  name,  fault  isolation  test 
( FIT) . 

Measurement  Terms:  In  the  case  studies  and  throughout  the 
diagnostic  literature,  there  are  numerous  approaches  for  improved 
measurement  of  diagnostics  performance.  The  focus  here  is  to  use 
relatively  common  operational  terms.  Though  the  terms  are  common, 
there  are  significant  differences  between  the  Services  and  various 
programs.  The  term  "false  alarm"  is  an  example  of  the  definition 
problem. 

The  Navy  typically  uses  the  term  false  alarm  to  refer  to 
those  BIT  indications  to  the  aircrew  which  results  in  an 
organizational  level  action  of  A-799  (no  defect),  plus  those 
maintenance  actions  which  result  in  an  organization  (o)  level 
weapons  replaceable  assembly  (WRA)  removal  followed  by  an  A-799 
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in  the  intermediate  (I)  level  shop.  It  is,  thus,  the  ratio  of  O&I 
level  A-799s  to  the  total  BIT  detected  maintenance  actions.  It 
is  a  BIT  maintenance  effectiveness  number  analogous  to  the  Air 
Force  terms  O  level  CND  and  I  level  bench-check  serviceable  (BCS) 
or  retest  OK  (RTOK).  The  false-alarm  denominator  is  total  BIT 
detected  maintenance  actions. 

The  Air  Force’s  typical  use  of  the  term,  false-alarm,  refers 
to  those  BIT  indications  to  the  aircrew  which  are  unaccompanied 


by  system  degradation,  not  debriefed  as  problems,  and  they  do  not 
become  maintenance  actions.  it  is  thus  a  measure  of  the  quality 
of  BIT  data  presented  to  the  aircrew.  The  false-alarm  denominator 
is  the  total  BIT  fault  indications. 


I 


In  a  contractual  sense,  false-alarm  rate  can  be  used  in 
either  of  the  ways  indicated  above.  However,  it  has  also  been 
interpreted  in  a  third  sense;  that  is,  as  the  ratio  of  incorrect 
BIT  indications  to  total  BIT  inquiries.  The  denominator  in  this 


case  is  the  total  number  of  times  BIT  measurements  are  made 
whether  they  result  in  an  indication  of  a  fault  or  not. 

Because  of  the  significant  differences  in  the  definitions  of 
false  alarm,  it  is  important  to  understand  which  is  being  used  in 


a  specific  application.  In  the  case  of  the  F/A-18  (APG-65  radar), 
the  Navy  operational  testing  threshold  for  false  alarm  was  20 
percent,  which  is  a  very  stringent  requirement.  Early  operational 
measurement  of  the  radar  false-alarm  rate  ranged  from  69.5  to 
79.2  percent.  The  contractual  requirement  was  for  a  false-alarm 
rate  of  _<  2  percent. 

The  Air  Force  definition  of  false  alarm  was  not  levied  as  a 
requirement  on  the  APG-66  radar,  nor  was  it  measured  duriny 
service  test  and  field  operation  due  to  a  lack  of  recording 


capability  to  capture  all  the  in-flight  BIT  indications.  On  two 
other  complex  avionics  systems  (E-3A  Surveillance  Radar  and  B-52  OAS) 
which  had  a  recording  capability,  the  false-alarm  rate  was  on  the 
order  of  70  to  80  percent. 


The  third  definition  is  not  applicable  to  the  three  radar 
examples  in  this  case  study.  However,  an  unrelated  example 
should  help  to  understand  what  the  order  of  magnitude  for  this 
definition  should  be.  In  this  program,  the  required  false-alarm 
rate  is  2  percent.  The  denominator  is  the  number  of  times  BIT 
measures  a  parameter.  This  particular  BIT  system  measures  12,000 
parameters  during  pre-flight  and  every  30  minutes  during  flight. 

In  a  4.5  hour  flight,  there  are  12,000  X  10  or  120,000  measurements 
made.  A  2  percent  FA  rate  would  mean  that  there  could  be  as 
many  as  2,400  false  indications.  Operationally,  this  does  not 
sound  like  a  constraint  at  all  when  one  considers  the  aircrew 


having  to  deal  with  this  much  false  information  and  distinguish 
it  from  true  fault  indications.  Instead  of  2  percent,  the  require¬ 
ment  should  more  properly  be  .002  percent  or  .0002  percent. 

In  summary,  false  alarms  by  any  definition  have  an  operational 
impact.  It  is  important  to  specify  exactly  what  definition  of 
false  alarm  is  being  used  on  a  specific  program.  Depending  on 
the  definition,  a  20  percent  FA  rate  can  be  very  stringent  and  a 
2  percent  FA  rate  can  be  very  loose. 

In  this  report,  false  alarm  will  be  used  to  refer  to  indications 
to  the  crew  which  do  not  become  maintenance  actions,  except  where 
it  specifically  applies  to  the  F/A-18. 

Cannot  duplicate  ( CND)  will  be  used  to  refer  to  those 
organizational  level  maintenance  actions  in  which  no  repair  action 
is  taken  because  the  reported  problem  could  not  be  duplicated. 

There  are  instances  where  a  problem  cannot  be  duplicated  but  a 
unit  is  removed  because  it  is  suspected  to  be  a  possible  cause 
of  the  problem.  Though  one  could  make  a  case  that  this  should 
be  called  a  CND,  it  is  not  reported  as  such  in  the  data  system 
and  will  not  be  considered  a  CND  in  this  discussion. 

Bench-checked  serviceable  (BCS)  will  refer  to  those  units 
removed  for  cause  which  subsequently  check  serviceable  at  the 
intermediate  level  of  maintenance  (Navy:  A-799  at  level  2). 
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Retest  OK  ( RTOK )  is  sometimes  used  to  describe  what  is 
defined  above  as  BCS.  However,  in  this  discussion,  it  will  apply 
to  a  unit  which  checks  serviceable  when  sent  to  the  depot  level 
for  repair. 

B.  CURRENT  PRACTICE  AND  SUCCESS 

For  the  past  decade  the  typical  diagnostics  requirements  for 
complex  avionics  have  taken  the  form  of  90-98  percent  fault  detection 
90-98  percent  fault  isolation,  and  1  to  2  percent  false  alarms, 
though  the  latter  is  not  always  specified.  Design  techniques  and 
contractual  demonstrations  have  evolved  to  meet  these  requirements. 
However,  though  they  have  been  achieved  in  a  contractual  sense, 
they  typically  are  not  met  during  early  operational  testing. 

Figure  E-l  shows  the  requirement  and  contractual  demonstration 
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FIGURE  E-l.  Radar  Diagnostics:  Success  or  Failure 
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results  of  three  airborne  radars.  Though  the  demonstration  require 
ment  was  substantially  achieved  in  all  three  cases,  field  testing 
resulted  in  a  significant  shortfall.  These  cases  are  not  unique. 
Future  programs  should  be  built  on  understanding  how  to  define, 
specify,  design,  demonstrate  and  mature  a  diagnostics  capability 
which  is  effective  during  operational  employment  by  Service. 

1 .  Requ irements 

The  current  built-in-test  requirement  development  process  has 
been  the  concern  of  the  military  as  well  as  contractors.  To 
properly  develop  requirements  so  that  the  problems  experienced 
with  BIT  (namely  insufficient  detection,  improper  fault  isolation 
and  excessive  false  alarms)  can  be  eliminated  from  designs  requires 
guidance  in  identifying  BIT  design  drivers  and  formulating  design 
decisions  to  preclude  as  many  of  the  identified  problems  and 
concerns  as  possible. 

The  problems  were  addressed  in  the  OSD,  February  1981  BIT 
Requirements  Workshop  (reported  in  IDA  Paper  P-1600,  August  1981). 
This  government/industry  workshop  was  sponsored  by  the  office 
of  the  Assistant  Secretary  of  Defense  (MRA&L)  to  assess  progress 
and  problems  in  specifying,  testing  and  evaluating  BIT  in  complex 
electronics  equipment.  The  following  is  an  overview  of  the 
problems  and  positions  expressed  in  the  report  which  have  appli¬ 
cation  to  the  development  of  design  requirements. 

•  BIT-equipped  electronic  subsystems  being  introduced 
into  the  field  today  are  universally  not  meeting  the 
diagnostic  specifications  which  are  generally  in  the 
range  of  90  to  95  percent  probability  of  automatic 
(or  semi-automatic)  fault  detection  and  isolation. 

•  20  to  40  percent  of  items  replaced  are  found  to  have  no 
failure  when  tested  at  the  Intermediate  or  Depot  level. 

•  BIT,  in  general,  is  not  designed  to  detect  all  failures. 
Consequently,  manual  troubleshooting  is  required  to  aug¬ 
ment  automatic  BIT  capability. 
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•  BIT  mechanization  has  not  yet  advanced  to  the  point 
where  highly  skilled  technicians  are  no  longer  necessary. 

•  BIT  design  during  system  development  is  not  as  visible 
to  program  management  as  is  electrical  and  mechanical 
design . 

•  BIT  contractual  specification  requirements  are  open 
to  a  wide  range  of  interpretations. 

•  Early  assessment  of  field  operational  performance  is 
very  difficult  because  of  incomplete  software  and 
interactions  between  operational  and  maintenance 
personnel,  test  equipment  and  technical  manuals. 

•  Adequate  test  time  (or  test  articles)  generally  has 
not  been  set  aside  to  develop  BIT  in  complex  systems. 

•  Contractual  BIT  laboratory  demonstration  tests 
(MIL-STD— 471 )  do  not  provide  reliable  predictions 
of  BIT  performance  in  the  field. 

•  About  two  years  of  field  operations  with  dedicated 
field  personnel  and  closed-loop  data  systems  have 
been  found  necessary  to  mature  BIT  in  complex 
systems. 

The  body  of  the  IDA  report  presents  papers  and  panel  dis¬ 
cussions  concerning  these  topics.  Its  primary  purpose  was  to 
document  the  lessons  learned  and  associated  recommendations 
to  improve  the  manner  in  which  BIT  is  specified,  and  its  design 
progress  monitored  and  evaluated,  both  analytically  and 
in  tests  and  demonstrations.  The  three  basic  issues  of  fault 
detection/isolation,  false  alarms,  and  design  growth  can  be 
identified  as  underlying  all  the  concerns  expressed  in  the  paper. 

The  fact  that  fault  detection/isolation  has  been  classically 
equated  to  probabilities  in  terms  of  failure  rates  of  components, 
components  count  or  failure  rates  of  functions,  and  the  fact 
that  the  term  confidence  in  fault  detection/isolation  can  be 
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interpreted  either  as  a  statistical  boundary  or  a  limit  on 
false  results,  illustrates  that  BIT  requirements  are  also  open 
to  interpretation. 

False  Detection/Isolation;  Failure  to  detect  faults  of 
which  either  the  operator  or  maintenance  crew  become  aware  re¬ 
sults  in  mistrust  of  the  BIT  and  heavier  reliance  on  squawks. 
Squawks  are  not  always  related  to  failures;  they  may  be  in 
response  to  normal,  but  unexpected  performance  stemming  from 
inexperience,  inadequate  training/technical  orders,  or  anomalous 
events  that  occur  during  otherwise  normal  operation.  The  result 
is  increased  maintenance  actions,  spares  depletion  and  decreased 
operational  availability.  Erroneous  fault  isolation  or  large 
ambiguity  groups  also  lead  to  spares  depletion,  excess  mainten¬ 
ance,  the  need  for  higher  skills  and  mistrust  of  the  BIT  results 
Figure  E-2  is  an  extract  from  the  IDA  report  outlining  past 
Air  Force  history  of  BIT  requirements  versus  results  of  design 
analyses.  The  analyses  were  based  on  either  field  data  or  study 
reports,  and  they  draw  attention  to  a  generally  large  disparity 
between  specified  and  attained  values.  Many  of  the  equipments 
were  thought  to  have  met  the  requirements  as  interpreted  by 
the  contractor.  The  Air  Force  Operational  Test  and  Evaluation 
Center  focused  on  this  disparity  in  three  operational  tests; 

The  E-3A,  F-16 ,  and  EF-111A.  The  resulting  study  identified 
the  problems  as  basic  to  the  prevailing  diagnostic  theory.  As 
shown  in  Fig.  E-3,  90  percent  fault  detection  can  be  attained 
by  checking  units  1  through  5  only,  since  each  individually 
contributes  18  percent  to  the  total  failure  rate.  Similarly, 
only  units  1  through  4  need  be  isolated  to  meet  an  80  percent 
isolation  capability.  The  specifications  have  been  met,  but; 

•  Units  6  through  10  may  be  more  mission  critical 
than  1  through  5  and  the  relative  distribution 
of  failure  rates  may  differ  from  the  prediction. 

•  Fault  detection  for  units  6  through  10  and  fault 
isolation  for  units  5  through  10,  must  be  performed 
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manually,  probably  based  on  operator  squawks  or 
comprehensive  ground  tests. 

•  The  BIT  design  could  have  left  the  most  difficult 
testing  and  fault  isolation  to  be  done  manually, 
which  is  contrary  to  the  objectives  of  automatic 
self-testing . 

•  Wiring  and  connectors  have  not  been  addressed. 

•  Undesirable  indications  (which  lead  to  false 
alarms,  cannot  duplicate  and  retest  ok  condi¬ 
tions)  have  not  been  designed  out. 

Though  there  was  no  general  consensus  as  to  the  exact  process 
of  establishing  diagnostic  requirements  in  terms  of  performance 
levels,  it  was  agreed  and  recommended  that  diagnostics  must  result 
in  meeting  operational  constraints,  and  that  these  should  be  stated 
in  terms  of: 

•  Identification  of  safety  and  mission  critical 
functions  including  maintenance  turnaround  time. 

•  False-alarm  rate. 

•  Constraints  on  the  use  test  equipment. 

•  Programmed  manpower  and  skill  levels. 

False  Alarm:  False  alarms  are  one  of  the  most  serious  problems 
encountered  with  BIT.  They  cause  false  mission  aborts,  inflight 
indications  where  organizational  level  maintenance  testing  cannot 
duplicate  the  problem  (CND),  or  false  removals  which  retest  ok 
( RTOK )  at  higher  maintenance  levels.  This  results  in  excessive 
shoploading  and  flightline  maintenance,  which  in  turn  decreases 
operational  availability  and  increases  support  costs.  In  the  long 
run,  false  alarms  result  in  the  operator  ignoring  BIT  indications, 
which  can  lead  to  mission  safety  problems  and  excessive  LRU  swapping 
which  in  turn  leads  to  maintenance-induced  failures. 

The  Air  Force  has  expended  rubstantial  effort  and  resources 
to  determine  how  to  eliminate  false  alarms  from  BIT  systems.  Of 
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particular  note  is  an  RADC  (RADC  Report  TR-81-220)  study  titled, 
"Analysis  of  Built-In-Test  False  Alarm  Conditions".  An  excerpt 
from  the  statement  of  work  is  as  follows: 

"The  negative  impact  of  false  alarms  on 
maintenance  policies  and  support  costs  has 
been  documented  on  a  wide  range  of  systems. 

The  extent  of  these  false  alarms  contributes 
to  the  expenditure  of  excessive  maintenance 
resources  such  as  manpower,  support  equip¬ 
ment,  and  logistic  supplies.  BIT  systems 
that  experience  high  levels  of  false  alarms 
may  be  rendered  ineffective  due  to  the 
lack  of  confidence  in  the  integrity  of 
the  failure  diagnostic  information. 

False  alarms  can  seriously  degrade  the 
mission  effectiveness  of  systems  that 
incorporate  BIT  to  perform  system  moni¬ 
toring  functions.  Erroneous  indications 
of  a  system's  capability  may  result  in  an 
unnecessary  mission  abort  depending  on 
the  criticality  of  the  system  under  test." 

"The  basis  of  false  alarm  conditions 
rests  in  unanticipated  design  deficiencies. 

Providing  designers  with  guidelines  to  anti¬ 
cipate  and  remedy  these  deficiencies  will 
result  in  a  BIT  end  product  with  high  level 
of  operator  confidence  in  the  validity  of  the 
test  results.  Prediction  factors  will  provide 
insight  concerning  and  allow  for  the  structur¬ 
ing  of  maintenance  policies  to  minimize  the 
impact  of  false  alarm  condi tions. " 


The  study  clearly  recognizes  the  impact  of  anomalous 
system  events  and  intermittent  faults  on  BIT  false  alarm  rates. 
There  are  four  possible  causes  of  a  BIT  fault  indication: 


1.  A  valid  fault 

2.  An  intermittent 

3.  A  momentary  anomaly 

4.  A  false  alarm. 


Historically,  because  of  a  lack  of  data,  the  last  three  have  all 
been  categorized  as  erroneous  indications  or  false  alarms. 
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Complex  military  systems  have  a  tendency  to  exhibit  momen¬ 
tary  anomalies  unrelated  to  the  presence  of  a  fault.  Most  system 
anomalies  do  not  indicate  failure  events  causing  loss  of  capability 
or  requiring  maintenance  action. 

To  quote  from  the  RADC  study: 

"BIT  must  be  designed  to  identify  and  accept 
'normal'  anomalies  without  generating  a 
failure  indication  (false  alarm)." 

It  is  very  difficult  to  differentiate  between  the  symptoms  of 
false  alarms  and  those  of  intermittent  faults.  Both  are  inherently 
transient  in  nature  and  extremely  difficult  (but  not  impossible)  to 
isolate.  The  solution  to  the  false-alarm  problem  should  not  be 
allowed  to  mask  the  problem  of  detecting  and  isolating  intermittent 
faults,  and  the  specified  requirements  must  ensure  that  it  is  clear 
that  thresholds  of  detection  cannot  be  lowered  to  mask  the  problem. 

The  selection  of  fault  detection  logic  and  the  associated 
thresholds  must  receive  a  great  deal  of  attention  during  the  design 
phase,  but  the  actual  fault  detection  threshold  settings  to  be 
used  after  operational  deployment  must  be  matched  to  real-world 
performance,  and  cannot  be  firmly  developed  until  flight  tests  and 
possibly  early  deployment.  Therefore,  requirements  must  also  address 
and  permit  BIT  growth  during  the  operational  phase  of  a  program. 

The  RADC  study  recognized  that  past  specifications  for  BIT 
performance  have  been  poorly  defined  at  best,  and  have  ignored 
the  problems  of  anomalies  and  intermittent  faults  which  emphasizes 
the  care  needed  in  developing  contractual  requirements. 

It  is  clear  from  the  studies  referenced  above  and  this  case 
study  activity,  that  the  traditional  methods  of  simply  specifying 
automated  detection  and  isolation  rates  provide  an  inadequate 
foundation  on  which  to  design  effective  operational  diagnostics. 
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2.  Design 


Current  diagnostics  design  practices  were  built  on  the 
existing  (or  evolving)  practices  of  the  reliability,  maintain¬ 
ability  and  logistics  disciplines.  Identification  of  the  failure 
modes  to  be  detected  was  based  upon  the  existing  reliability 
analyses,  i.e.,  the  reliability  prediction  and  the  failure  mode  and 
effects  analysis. 

These  reliability  techniques  originally  addressed  only  hard 
or  catastrophic  failure  modes.  Today's  complex  electronics  also 
exhibit  many  subtle  forms  of  out-of-tolerance  performance  that 
are  transient  and  result  from  timing  incompatibilities,  unforeseen 
processing  and  environments,  or  operational  and  diagnostic  soft¬ 
ware  problems.  The  result  is  that  diagnostic  analyses  based  upon 
less  than  comprehensive  FMEAs  produce  incomplete  estimates  of  the 
design's  diagnostic  capability. 

More  modern  approaches  have  been  evolving  and  do  have  the 
potential  to  improve  diagnostic  design.  These  include  management 
emphasis  on  diagnostics,  architectural  schemes  which  facilitate 
fault  detection  and  isolation,  earlier  and  more  effective  logistics 
support  analysis  to  influence  design  trades,  and  emerging  techniques 
for  testability  analysis.  As  an  example,  in  the  FIREFINDER  Program, 
BIT  design  was  accomplished  concurrently  with  the  radar  system 
design.  Test  point  selection  was  made  in  system  layout.  Computer 
simulation  of  the  signal  processor  was  made  to  establish  optimum 
implementation  of  BIT. 

Advancing  technology  in  multiplex  busing,  microprocessors  and 
distributed  processing  permitted  architectures  where  individual 
subsystems  could  be  self-contained  with  respect  to  fault  detection 
and  isolation.  Improved  partitioning  of  functions,  self-contained 
BIT  integration  in  subsystem  design,  and  simple  subsystem  interfaces 
greatly  facilitated  fault  isolation. 

Logistics  support  analysis,  closely  linked  to  the  system 
engineering  process,  served  to  focus  designers  on  the  operational 
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support  implications  of  design  trades.  On  the  F/A-18,  subsystem 
design  engineers  were  responsible,  not  only  for  the  performance 
characteristics,  but  for  R&M  as  well. 

Testability  analysis  techniques,  though  immature,  have  had 
some  impact  on  achieving  diagnostic  capability.  These  techniques 
aid  in  the  optimum  placement  of  test  points,  assessing  the  degree 
of  physical/functional/electrical  partitioning  achieved  in  a  design, 
allocating  hardware  and  software  to  perform  BIT,  setting  test 
tolerances  to  balance  between  failure  detection  and  false  alarms, 
and  providing  a  disciplined  structure  for  achieving  vertical  test¬ 
ability. 

The  overall  result  of  these  evolving  design  approaches  has 
been  a  significant  improvement  in  diagnostic  capability  over  pre¬ 
vious  weapon  systems,  such  as  the  F-4,  F-lll,  and  the  initial  F-15. 
Though  false-alarm,  CND  and  RTOK  problems  have  not  been  totally 
overcome  in  current  programs,  complex  electronic  systems,  such  as 
the  F-16  and  F/A-18,  would  be  virtually  impossible  to  maintain 
without  the  improved  design  approaches. 

In  current  complex  systems,  software  development  (particularly 
for  diagnostics)  has  become  a  difficult  problem.  A  hypothetical 
example  illustrates  why.  At  the  front  end  of  the  design  activity, 
detailed  requirements  tend  to  be  unfixed,  which  forces  the  hard 
tasks  to  get  relegated  to  software  to  be  resolved  later  in  design 
where  they  can  presumably  be  resolved  with  little  hardware  impact. 
The  first  design  priority  is  designing  the  hardware  to  perform,  and 
the  development  of  diagnostics  software  lags  performance  implemen¬ 
tation.  The  software  requirements  baseline  may  not  even  be  fully 
defined  by  the  time  testing  starts.  The  testing  of  diagnostic 
software,  if  it  is  done  at  all,  normally  has  less  emphasis  than 
mission-critical  software.  The  maintainability  demonstration 
employs  the  diagnostic  software,  but  in  a  highly  controlled  environ¬ 
ment  where  known  hard  failures  (not  necer_.  i ly  the  causes  of 
false  alarms)  are  inserted.  When  operational  testing  starts,  the 
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software  is  immature,  but  in  addition,  the  use  patterns  and  environ¬ 
ment  change  significantly  from  those  of  the  demonstration.  At  this 
point  unanticipated  faults,  previously  untested  software  paths,  and 
unexpected  occurrences  result  in  undetected  failures  and  false  alarms. 
With  proper  planning,  this  experience  can  be  used  to  adjust  thresholds 
voting  schemes,  and  alarm  decision  algorithms.  What  is  needed  then 
is  software  performance  data,  software  engineering  expertise  and  a 
rigorous  system  for  tracking  and  updating  software  configuration. 

Better  software  design  and  management  techniques  would  be 
useful  in  maturing  the  Software  earlier. 

3.  DeveloDment  and  Demonstration  Test 


Diagnostic  testing  has  included  development  tests  and 
verification  or  demonstration  tests.  Development  testing  has 
been  performed  primarily  using  either  fault  insertion  exercises 
or  using  other  tests  as  targets  of  opportunity. 

Fault  insertion  exercises  were  used  for  BIT  development  on  the 
F-16  in  general  and  the  APG-66  radar  in  particular.  Since  system 
requirements  forced  the  BIT  to  be  self-contained  within  subsystems, 
effective  subsystem  BIT  testing  was  possible.  Prior  to  formal 
demonstration,  the,  radar  contractor  inserted  over  1450  faults  and 
used  the  results  to  improve  detection  and  isolation  capabilities. 
The  activity  paid  off  in  the  formal  demonstration  where  94  percent 
fault  detection  and  98  percent  fault  isolation  were  demonstrated 
for  150  inserted  faults. 

Besides  fault  insertion  development  exercises,  contractors 
have  effectively  used  other  activities  as  targets  of  opportunity 
to  identify  BIT  problems.  Though  not  always  efficient,  these 
tests  can  provide  additional  opportunities  to  assess  how  the  BIT 
performs  for  occurrences  other  than  known  inserted  faults.  In  one 
example  cited  during  the  study,  actual  failed  items  were  collected 
from  throughout  the  development  and  production  activities  and  were 
processed  through  a  system  integration  laboratory  to  assess  the 
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reaction  of  the  diagnostics  to  actual  failures  which  had  occurred 
naturally. 

In  the  F-l 5  APG-63  radar,  the  three  phases  of  reliability 
qualification  and  production  reliability  testing  resulted  in  38 
software  changes,  nearly  all  of  which  were  BIT  improvements.  The 
F-16  employed  BIT  during  reliability  testing  to  verify  that  it  would 
produce  consistent  indications  with  factory  acceptance  test  proce¬ 
dures. 

Environmental  testing  can  provide  opportunities  for  identifying 
BIT  problems,  changing  tolerances,  and  modifying  voting  criteria. 

It  should  not  be  used  as  a  demonstration  tool,  in  part  because  of 
the  limited  sample  of  serial  number  assets  and  fault  occurrence  as 
well  as  the  type  of  faults  encountered  (infant  mortalities  and 
design  deficiencies  which  are  likely  to  be  fixed).  The  use  of 
reliability  tests  for  BIT  maturation  should  be  encouraged. 

Fault  insertion  and  maintainability  demonstration  tests 
usually  result  in  a  successful  verification  of  BIT  FD/FI  require¬ 
ments.  Since  these  demonstrations  are  usually  followed  by 
significant  BIT  problems  in  early  field  testing,  one  school  of 
thought  is  to  do  away  with  these  demonstrations  as  a  waste  of 
time  and  money.  It  is  necessary  to  review  the  benefits  and  limi¬ 
tations  of  such  demonstrations  before  dismissing  them. 

There  are  benefits  to  FD/FI  demonstrations  as  they  are  currently 
conducted.  They  force  the  diagnostic  design  to  mature  sufficiently 
by  a  required  time  in  the  development  cycle  so  as  to  pass  the 
contractual  demonstration  requirement.  Secondly,  they  force  a 
disciplined  and  structured  consideration  of  BIT  characteristics 
(FD/FI)  which  is  useful  operationally  as  well  as  being  measurable 
in  a  contractual  sense.  Without  this  motivation,  one  might  anticipate 
the  possibility  that  diagnostic  design  would  lag  performance  even 
further.  Thus,  without  some  better  motivation,  they  should  be 
considered  necessary. 
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Though  necessary,  they  are  not  as  effective  as  they  could  be. 
Fault  insertion  tests  have  some  major  flaws  in  the  way  faults  are 
selected.  The  candidate  fault  list  is  only  a  small  sub-set  of 
the  total  known  faults  which  could  be  introduced.  Since  it  is 
based  on  predicted  failure  modes  and  frequencies,  total  population 
of  interest  is  only  as  good  as  the  FMEA.  Since  the  FMEA  does  not 
adequately  address  false  alarms  and  their  causes,  the  demonstrations 
are  not  likely  to  encounter  many  of  the  diagnostic  problems  observed 
in  the  field.  The  fault  selection  process  is  also  influenced  by 
the  desire  to  not  insert  faults  that  would  damage  the  equipment. 
Therefore,  the  chosen  faults  are  simple  to  install;  e.g.,  shorting 
something  rather  than  unsoldering,  or  avoiding  inducing  faults  in 
expensive  parts.  This  tendency  can  result  in  an  unrealistic  bias 
toward  the  set  of  simple  failures,  avoiding  those  that  are  difficult 
and  expensive  to  prepare. 

While  fault  insertion  demonstrations  co  not  provide  a  complete 
answer,  they  have  the  potential  for  expansion  to  a  level  where 
many  diagnostic  problems  can  be  resolved  during  full-scale  develop¬ 
ment  and  before  the  system  is  deployed  in  the  operational  environ¬ 
ment. 


» 


4 .  Operational  Test/Maturation 


Early  operational  testing  on  the  F-15,  F-16  and  F/A-18  radars 
(and  many  other  subsystems  and  programs)  has  been  characterized 
by  the  discovery  of  high  BIT  false-alarm  rate2;  (including  high 
"cannot  duplicate"  and  "bench  check  serviceable"  rates),  diagnostic 
system  immaturity,  numerous  software  iterations,  long  trouble¬ 
shooting  times  and  lower  rates  of  fault  detection  and  isolation 
than  had  been  expected  (based  on  the  specifications  and  main¬ 
tainability  demonstrations).  Depending  on  the  particular  program 
and  the  severity  of  the  problems,  a  variety  of  actions  were  taken 
to  recover.  In  some  cases,  an  ad  hoc  logistics  support  structure 
had  to  be  created  and  overlaid  on  the  planned  logistics  support 
concept.  Workarounds  were  developed  to  compensate  for  BIT  short¬ 
falls.  These  workarounds  frequently  included  special  test  aids 
(such  as  troubleshooting  fault  trees,  breakout  boxes,  data  bus 
monitors,  and  memory  inspect  devices)  as  well  as  contractor 
engineering  support  to  compensate  for  inadequate  training  and 
technical  data.  The  workarounds  typically  persisted  until  the 
BIT  shortfalls  were  corrected  or  until  the  workarounds  became 
institutionalized  in  the  support  structure.  The  E-3A  surveillance 
radar,  for  example,  had  to  pursue  both  correction  of  BIT  shortfalls 
and  institutionalizing  workarounds.  The  F-16  went  through  a 
deliberate  process  of  maturation  to  improve  the  built-in  test. 

The  F/A-18  is  going  through  a  similar  planned  maturation  program 
for  diagnostics. 

One  of  the  more  severe  impacts  of  BIT  shortfalls  is  the  lack 
of  confidence  which  operational  crews  and  maintenance  technicians 
develop  toward  the  BIT.  This  low  confidence  leads  to  poor  mainten¬ 
ance  troubleshooting  practices  which,  once  created,  are  difficult 
to  overcome. 

As  discussed  earlier,  the  effectiveness  of  demonstration  testing 
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is  limited  by  the  manner  in  which  faults  are  selected  and  the  prac¬ 
tical  bounds  of  time,  number  of  test  samples  and  cost.  The  field 
environment  provides  a  realistic  opportunity  for  identifying  and 
maturing  BIT.  The  implication  for  design  is  that  the  BIT  should 
be  readily  adjustable.  Improvements  should  be  able  to  be  incor¬ 
porated  in  software  with  minimal  hardware  impact.  Contractual 
techniques  need  to  be  developed  to  plan  for  field  maturation  as 
a  positive  design  technique  (an  extension  of  the  design  activity) 
rather  than  having  the  stigma  of  design  patching. 


a.  F-16  ( APG-66  Radar).  The  APG-66  diagnostics  growth  and 
maturation  period  spanned  44  months  (May  1977-Dec.  1980)  including 
a  large  overlap  of  full-scale  development  and  production  phase 
activities  (see  Fig.  E-4).  The  eight  configuration  iterations 
from  May  1977  to  Dec.  1978  (20  months)  were  largely  associated 
with  growing  from  45  self-tests  with  no  BIT  to  the  full  108  self 
test/BIT  capability.  In  Jan.  1979,  the  first  squadron  was  acti¬ 
vated  at  Hill  AFB.  Pilot  usage,  aircraft  interface  and  inter¬ 
mittent  problems  were  an  early  indicator  of  the  need  for  maturing 
the  diagnostic  capability.  The  maturation  effort  continued  from 
Jan.  1979  to  Dec.  1980  (24  months).  Incremental  changes  were 
consolidated  in  ECP-331  (B++  radar)  which  was  incorporated  in 
production  radar  and  retrofit  in  1980.  This  ECP  significantly 
reduced  the  occurrence  of  faults  and  CNDs  (Fig.  E-5  and  E-6). 
Figure  E-7  is  another  indicator  of  the  improvement  in  detection, 
isolation  and  CND  rates  from  development  flight  testing  in  1978 
to  the  FOT&E  II.  Though  there  was  a  significant  improvement, 
about  47  percent  of  the  total  FOT&E  write-ups  were  in  the  category 
of  known  engineering  deficiencies  and  were  not  considered  in  the 
BIT  statistics.  This  category  declined  significantly  throughout 
the  test  (from  .3  per  sortie  to  .03  per  sortie)  as  improvements 
were  introduced. 
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FIGURE  E-5.  Radar  MUX-BUS  Avionics  Maintenance  Fault  List 
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FIGURE  E-6. 


B++  Radar  Modification  Effects 
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b.  F/A-18  (APG-65  Radar).  The  F/A-18  has  gone  through  a 
series  of  tests  for  the  purpose  of  assessing  and  maturing  the 
diagnostics  capability  of  the  aircraft  systems.  The  event  which 
indicated  that  a  significant  maturation  effort  would  be  required 
was  the  IOT&E  conducted  by  the  Operational  Test  and  Evaluation 
Force.  Specific  radar  subsystem  data  were  not  presented  in  the 
quick-look  report.  However,  it  did  report: 

Avionics  BIT  is  incomplete  (82%) 

MMP  high  false  alarm  rate  (82%) 

High  initiated  BIT  false  alarm  rate  for  major  system 
failures  (56%) 

Low  WRA  BIT  isolation  validity  (56%) 

Radar  and  flight  control  systems  require  the  greatest 
level  of  effort  to  repair. 

Subsequently,  the  Navy  Bureau  of  Inspection  and  Survey 
(BIS)  conducted  trials  and  three  phases  of  a  BIT  assessment  program. 
Even  at  the  end  of  this  effort,  the  radar  was  still  identified  as 
one  of  the  major  problem  subsystems  (see  Fig.  E-8).  This  matura¬ 
tion  effort  is  planned  to  continue  in  several  steps.  Follow-on 
BIT  assessment  will  be  conducted  through  Sept.  1983  on  Block  7/8 
aircraft.  Mature  BIT  assessment  is  planned  for  Oct.  1983-Sept. 

1984  on  F-87  omnibus  ECP  (Block  11  aircraft).  A  concurrent  aircraft/ 
automatic  test  equipment  vertical  testability  assessment  started 
in  Mar.  1983  and  will  conclude  in  Sept.  1984.  The  total  effort 
(disregarding  IOT&E)  covers  34  months. 

At  the  start  of  field  testing,  the  diagnostics  capability 
was  not  only  immature,  many  tests  had  not  really  been  implemented. 
Throughout  the  maturation  process,  there  was  a  two-step  discipline 
for  declaring  the  capability  of  a  particular  test.  First,  the 
contractor  had  to  declare  the  test  functional  (i.e.,  essentially 
ready  to  be  assessed).  Secondly,  the  Navy  had  to  declare  the 
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FIGURE  E-8.  APG-65  Diagnostics  Maturation 


test  operational,  which  meant  the  test  had  to  meet  certain  false- 
alarm  and  performance  criteria.  Over  the  whole  period,  the 
various  test  codes  were  in  a  constant  state  of  flux  between  non¬ 
functional,  functional  but  not  operational,  and  operational. 

The  radar  BIT  improvements  are  listed  on  Fig.  E-9. 

c.  Electronic  Warfare  System.  Though  not  a  specific  case 
for  this  study,  an  electronic  warfare  system  provides  another 
example  of  an  effective  maturation  program.  Two  phases  of  FOT&E 
were  conducted  about  two  years  apart.  During  FOT&E  I,  FD  was  low 
at  62  percent  as  a  result  of  a  38  percent  CND  rate.  The  FI  rate 
was  71  percent,  largely  because  of  the  19.2  percent  RTOK  rate. 
These  indicators  were  symptoms  of  a  larger  suitability  problem 
which  included  deficiencies  in  reliability,  software,  technical 
data,  and  training.  The  program  subsequently  went  through  an 
intensive  maturation  phase  to  correct  these  problems.  The  diag¬ 
nostics  maturation  included  a  team  of  contractors  and  Air  Force 
technicians  at  the  plant  who  used  large  numbers  of  realistic  fail¬ 
ure  modes  to  devise  the  best  set  of  BIT  functions  and  technical 
procedures.  The  results  in  FOT&E  II  included  a  reduction  in  the 
CND  rate  from  38  to  24  percent,  an  improvement  in  the  FD  rate  from 
62  to  76  percent,  and  much  more  importantly,  a  dramatic  reduction 
in  the  RTOK  rate  from  19.2  to  3  percent. 

5.  Other  Analyses 

in  addition  to  the  case  studies  explicitly  conducted  for 
this  report,  other  analyses  which  are  pertinent  to  diagnostics 
were  reviewed. 


92/11-7 


L.  80%  BIT  matrix  unmasked, 
many  tests  not  functional, 
MMP  010  made  nonfunc¬ 
tional 

2.  ECP  079  reduced  hangups 
MMP  010  made  functional 
MMP  040-045,  052  made 
nonfunctional  unless 
accompanied  by  pilot 
squawk 

3.  New  tape  improved  FD/FI 
and  FA. 

95%  BIT  matrix  unmasked, 
MMP  040-045,  052  func¬ 
tional 

3  test  timing,  logic 
changes 

1.  ECP  044  improved  7  I  BIT 
test  points,  changed 
PBIT  criteria  for  FD, 
improved  FI  (trans, 

R-E,  antenna) 

5.  ECP  095  reduced  time  to 
test, 

improved  FI  accuracy 
reduced  source  of  test 
hang  ups 

'SW=sof tware ,  HW=hardware 


Ef  fectivit\ 


F-17,  Tape  100 


F-28,  tape  100A 


Character* 


F-28,  tape  101 


F-3  2 ,  tape  101A 


HW  &  SW 


F-37 ,  tape  101B 


FIGURE  E-9.  APG-65  Diagnostics  Maturation 
Radar  BIT  Improvements 
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a.  Causes  of  "Bench  Check  Serviceable"  (BCS). 
maintenance  data  collection  system  does  not  adequately  address 
the  reasons  for  a  high  BCS  rate.  Furthermore,  there  are  many 
error  sources  in  the  data  which  make  the  reported  BCS  rate  only 
a  tentative  indicator  of  actual  field  experience. 

In  addition  to  the  RADC  report  on  causes  of  false  alarms 
(referenced  in  para.  1  above),  two  other  studies  shed  some  light 
on  the  factors  which  cause  unnecessary  removals.  The  F-15  Avionics 
Maintenance  System  Task  was  conducted  by  McDonnell  Aircraft  from 
Aug.  1979  to  Apr.  1980  to  identify  the  sources  of  "Bench-Check 
Serviceable"  (BCS)  on  specific  avionics  Line  Replaceable  units 
(LRU).  The  purpose  was  to  recommend  means  to  eliminate  or  reduce 
BCS  loading  of  intermediate  level  automatic  test  equipment.  A 
field  study  was  conducted  at  Holloman  AFB  during  which  time  222 
LRUs  were  processed.  There  were  58  BCS  actions.  The  BCS  rate 
of  the  sample  was  either  25.3  percent  or  30.4  percent,  with  the 
uncertainty  created  by  data  recording  problems  (LRUs  coded  BCS 
which  were  in  fact  repaired,  and  units  where  the  action  taken 
was  unknown).  Figure  E-10  is  the  study  assessment  of  the  direct 
and  indirect  contributors  to  the  BCS  actions.  The  field  environ¬ 
ment  clearly  introduces  a  multitude  of  factors  which  can  produce 
a  BCS  action.  Furthermore,  the  maintenance  data  collection 
system  gives  little  insight  into  these  factors.  At  best,  it 
indicates  that  there  might  be  a  problem,  but  provides  no  infor¬ 
mation  with  which  to  identify  the  causes. 
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FACTORS 

NUMBER 
( Total 

OF 

58 

CASES 

BCS ) 

Direct 

Contributor 

Possible 

Contributor 

Technical  Orders 

6 

5 

Technician  Training 

— 

38 

Operator  Training 

5 

— 

Maint.  Mgmt.  and  Opns. 

26 

22 

Bui lt-in-Test 

— 

8 

Automatic  Test  Equipment 

— 

1 

Supply/Spaces 

— 

— 

Personnel  (Qty.  and  Skill) 

1 

— 

Flight  and  Shift  Schedules 

— 

9 

Reliability  ( Intermi ttents ) 

— 

15 

Figure  E-10.  F-15  Avionics,  Causes 

of  58  Bench  Check  Serviceables 
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The  second  effort  was  RADC-TR-83-2 ,  Jan.  1983,  entitled 
"Study  of  the  Causes  of  Unnecessary  Removals  of  Avionic  Equipment," 
performed  by  Hughes  Aircraft  Company.  The  field  survey  was 
conducted  at  12  bases  and  3  depots;  it  involved  six  selected 
avionic  equipments  used  on  ten  different  aircraft  platforms. 

The  average  unnecessary  removal  (UR)  rate  (same  as  BCS)  was 
32.7  percent.  The  UR  rate  on  the  same  equipment,  but  different 
platforms,  varied  from  zero  percent  to  80  percent.  For  the  same 
equipment  and  platform,  but  at  different  bases,  the  UR  rate 
varied  from  37  to  76  percent.  The  causes  of  URs  varied  signi¬ 
ficantly  with  the  specific  equipment,  as  one  would  expect.  The 
overall  sample  assessment  of  causes  is  listed  in  Fig.  E-ll. 


CAUSE  %  OCCURRENCE 

Ineffective  Built-In-Test  22 

Ineffective  or  Missing  Test  Equipment  18 

Ineffective  Supervision/Support  16 

Ineffective  Technical  Orders  13 

Inaccessibility  12 

Management  Directives  7 

Test  Equipment  Differences  7 

Inadequate  Skill  5 

Inadequate  Feedback  1 


FIGURE  E-ll.  RADC/Hughes  Assessment  of  Unnecessary 

Removal  Causes 
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These  studies  looked  at  equipment  which  was  fielded  and, 
in  general,  well  beyond  the  development  phase.  Several 
conclusions  can  be  drawn: 

a.  Once  a  system  is  fielded,  it  takes  a  special  field 
survey  effort  to  understand  (and  for  that  matter, 
accurately  estimate)  the  BCS  problem. 

b.  The  corrective  actions  required  to  resolve  BCS  problems 
in  fielded  systems  are  not  evident  without  such  a 
field  survey. 

c.  The  operational  environment  introduces  a  variety 
of  cause  factors  which  are  not  well  understood  or 
anticipated  in  the  design  process. 

A  possible  implication  for  design  is  that  the  more  effective 
the  front  end  design  and  maturation  processes  are  to  reduce 
the  sources  of  ambiguous  indications,  the  lower  the  field  BCS 
rate  will  be.  The  study  results  also  suggest  that  an  integrated 
approach  to  diagnostics  may  be  useful.  Maintenance  policy, 
technical  data,  training  and  personnel  skills  are  all  factors 
which  interact  with  BIT,  ATE,  and  vertical  testability  to 
produce  an  operational  maintenance  capability. 
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One  of  the  specific  LRUs  in  the  UR  study  was  the  radar  data 
processor  (WUC  74FQ0)  from  the  APG-63  radar.  508  removals  occurred 
and  266  (or  52  percent)  were  considered  unnecessary.  Ineffective 
built-in-test  was  judged  the  most  frequent  UR  cause  (45  percent). 
The  breakout  by  base  and  cause  is  indicated  in  Fig.  E-12. 


O  LEVEL 

LRUS  REMOVED 

I  LEVEL 

TOTAL  URS 

UR 

RATE  (%) 

125 

95 

76 

193 

90 

47 

157 

58 

37 

33 

23 

70 

508 

266 

52% 

BASE 

1 

2 

3 

4 

TOTAL 


CAUSE 


Ineffective  Built-In-Test 
Test  Equipment  Differences 
Skill  Level  inadequate 
Inadequate  Supervision/Support 


URs 


119.8 

66.8 

30 

49.3 

266 


CASE  RATE  (%) 


FIGURE  E-12.  APG-63  RDP  Unnecessary  Removals 

And  Causes 
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Other  sections  have 


Impact  of  Diagnostic  Capability, 
discussed  the  problems  associated  with  defining,  specifying,  desig¬ 
ning,  demonstrating,  and  collecting  field  data  on  diagnostics 
figures  of  merit. 

A  related  problem  is  the  criteria  associated  with  assessing 
a  figure  of  merit.  Is  95  percent  fault  detection  enough?  Does 
an  O-level  CND  rate  of  30  percent  really  impact  the  ability  to 
perform  the  mission?  Is  a  RTOK  (BCS)  rate  of  15  percent  a  more 
serious  problem  than  30  percent  CND?  Which  should  be  fixed?  The 
designer  and  the  operator  would  both  like  to  know  the  answers  to 
such  questions.  In  the  context  of  a  specific  weapon  program,  with 
a  significant  analytical  effort,  some  limited  answers  are  possible. 
However,  neither  the  tools  and  techniques  nor  the  data  for  answering 
these  questions  are  "on  the  shelf"  for  easy  assessment  and  trade-off 
of  various  levels  of  diagnostic  capability.  The  questions  are 
even  more  difficult  to  answer  in  the  generic  case.  The  following 
discussion  offers  some  suggestions  for  thinking  about  the  answer. 

Level  of  Automated  Fault  Detection  (FD)  -  The  answer  is  that 
enough  data  is  needed  to  tell  the  operator  that  a  reasonable  system 
capability  is  left  to  achieve  a  high  probability  of  mission 
success.  He  needs  to  know  the  capability  of  essential  mission 
functions.  If  the  system  is  fault  tolerant  and  redundant,  enough 
FD  to  automatically  reconfigure  the  system  must  be  provided  to 
accomplish  the  mission.  The  presentation  of  information  to  the 
aircrew  is  a  human  factors  trade-off.  Presenting  more  information 
than  the  operator  can  handle  ambiguously  could  just  as  easily 
jeopardize  mission  success  as  could  too  little  information.  In 
any  case,  the  indications  have  to  be  reliable  to  keep  a  high 
level  of  crew  confidence.  Quantifying  "enough  but  not  too  much" 
is  a  complex  task. 
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The  fault  detection  requirements  for  the  maintenance  technician 
are  somewhat  different.  Where  the  aircrew  may  need  information 
in  terms  of  seconds,  the  maintenance  crew  needs  are  more  in  terms 
of  minutes.  The  need  for  non-ambiguous  information  is  just  as  im¬ 
portant,  however. 

Fault  Isolation  (FI)  -  Provides  the  maintenance  personnel  the 
capability  to  deal  with  detections  and  effect  accurate,  rapid 
repair  actions.  Fault  detection  requirements  suggest  a  func¬ 
tionally-oriented  mechanization.  A  different  mechanization 
approach  may  be  necessary  to  isolate  discrete  units  for  replace¬ 
ment.  In  fault  tolerant  and  redundant  systems,  the  technician 
needs  insight  into  all  the  paths  which  need  repair.  Since  an 
airborne  fault  may  be  environmentally  mission  induced,  it  may  not 
be  possible  to  duplicate  on  the  ground.  In  this  case,  the  system 
will  have  to  remember  why  the  failure  occurred  so  that  the  mal¬ 
functioning  unit  can  be  replaced.  This  knowledge  must  be  preserved 
so  that  the  unit  can  be  repaired  in  a  field  shop  or  depot.  If 
not,  it  will  contribute  to  the  RTOK  rate. 

CND  Rate  -  The  CND  rate  can  be  viewed  on  the  basis  of  main¬ 
tenance  actions  or  on  the  basis  of  occurrences  per  sortie.  A  CND 
rate  of  50  percent  (1  out  of  every  2  maintenance  actions)  sounds 
excessive.  However,  a  50  percent  CND  rate  can  mean  dealing  with 
one  per  five  sorties  or  one  per  50  sorties.  Obviously,  the  occur¬ 
rence  on  a  sortie  basis  is  a  much  more  meaningful  parameter. 

The  mission  impacts  of  CNDs  are  twofold.  First,  the  end 
item  is  down  (unavailable  for  another  mission)  until  maintenance 
can  be  performed.  Secondly,  they  consume  maintenance  resources 
(available  manhours).  On  the  average,  a  CND  maintenance  action 
takes  less  time  to  repair  than  a  fix  action.  Figure  E-13  is  a 
comparison  of  the  airborne  radar  CND  rates,  frequencies,  and  associ¬ 
ated  repair  times.  Though  mean  times  to  repair  are  relatively  short, 
these  are  only  the  active  maintenance  times.  Every  time  a  mainten¬ 
ance  action  occurs,  whether  it  ultimately  turns  out  to  be  a  confirmed 
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failure  or  CND,  the  aircraft  also  experiences  the  downtime  associ¬ 
ated  with  the  scheduling  and  dispatching  of  a  maintenance  crew. 

In  the  case  of  the  F-18  radar,  the  awaiting  maintenance  factor  per 
maintenance  action  is  4.8  hours. 

In  a  surge  situation,  considerably  more  delay  time  might  be 
envisioned  because  of  the  additional  maintenance  demands  of  a 
high-sortie  generation  schedule. 

There  are  several  conclusions  to  be  drawn  from  Fig.  E-13. 

In  the  case  of  these  three  radars,  a  significant  number  of  in¬ 
flight  discrepancies  cannot  be  duplicated  on  the  ground.  Though 
cannot  duplicate  actions  probably  take  less  time  than  verified 
failures,  their  frequency,  time  to  repair,  and  the  fixed  time 
penalty  for  awaiting  maintenance  do  impact  aircraft  availability. 

One  can  judge  the  usefulness  of  CND  rate  versus  flying  hour 
frequency  as  measures  by  contrasting  the  F-15  and  F-16  radars. 

The  F-15  has  a  better  (lower)  CND  maintenance  action  percentage 
than  the  F-16,  but  it  also  has  far  more  CNDs  per  1,000  flying 
hours.  One  would  expect  the  impact  of  a  high  CND  rate  to  be  worse 
in  a  wartime  situation  than  in  a  peacetime  training  profile. 

Air  Force  data  collection  systems  suggest  that  CNDs  are  a  major 
maintenance  problem,  particularly  in  complex  electronic  subsystems 
such  as  fire  control.  On  a  survey  of  four  different  aircraft  fire 
control  systems,  CNDs  accounted  for  40  to  70  percent  of  the  fire 
control  maintenance  events  and  30  to  50  percent  of  the  fire  control 
maintenance  manhours. 

Bench  Check  Serviceable  (BCS)  Rate  -  Like  CNDs,  a  high  BCS  rate 
adversely  impacts  aircraft  availability  and  maintenance  resource 
consumption.  The  impact  on  aircraft  availability  is  less  direct, 
however.  It  only  begins  to  directly  impact  aircraft  availability 
when  all  available  spares  in  supply  have  been  issued.  Besides 
using  available  manhours  as  do  CNDs,  BCSs  also  consume  intermediate- 
level  test  station  time.  A  higher  induction  rate  of  units  which 
eventually  check  serviceable  also  means  a  higher  probability  for  a 


90/31-3 


backlog  of  units  awaiting  test  station  time.  Like  CNDs,  BCS  should 
not  just  be  viewed  as  a  rate  of  units  processed;  the  more  meaningful 
measure  of  the  impact  is  in  terms  of  occurrences  on  a  flying-hour 
basis. 

Figure  E-14  depicts  the  BCS  rates,  frequency  relative  to  flying 
hours,  and  time  and  manpower  factors.  Some  tentative  conclusions 
are  possible.  Nearly  one  of  every  four  units  processed  in  the 
shop  has  no  defect  found.  The  mean  time  to  repair,  though  longer 
than  0  level  CND  times,  is  only  a  part  of  the  penalty  for  incurring 
a  BCS.  A  typical  LRU  spends  3  to  5  days  in  the  base  repair  cycle 
pipeline,  unnecessary  removals  consume  man  hours  in  the  shop  as 
well  as  loading  the  test  stations  and  decreasing  the  availability 
of  spares.  It  is  difficult  to  estimate  the  direct  impact  of  BCSs 
on  aircraft  availability.  In  general,  a  high  BCS  rate  would  have 
a  greater  impact  on  aircraft  availability  during  a  surge  scenario. 

It  is  interesting  to  note  that  the  Air  Force  methodology  for 
computing  replenishment  spares  quantities  does  not  include  BCSs. 
Spares  are  replenished  only  on  the  basis  of  units  actually  repaired 
in  the  shop.  Thus  BCSs  do  not  increase  the  number  of  spares  in 
the  system,  but  they  do  increase  the  utilization  (and  outage)  of 
the  ones  which  are  in  the  system,  as  well  as  creating  the  need  for 
cannibalization  when  base  level  stocks  are  depleted. 

Repair  Times  -  Assessing  the  impact  of  BIT  on  repair  times  is 
straightforward  in  theory  but  difficult  in  practice.  Normal  main¬ 
tenance  data  collection  systems  do  not  distinguish  whether  trouble¬ 
shooting  (fault  isolation)  is  performed  through  BIT  (automated  or 
semi -automated )  or  through  manual  techniques  or  a  combination  of 
the  two.  Results  from  a  few  special  tests  provide  some  insight  into 
the  contribution  of  BIT  to  the  overall  mean  time  to  repair.  In 
one  Air  Force  operational  test  of  an  electronic  warfare  system, 
such  measurements  were  made.  The  Mean-Time-to-Repair  (MTTR)  model 
was  partitioned  into  five  parts:  Set-up  of  Ground  Support  Equipment 
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(or  AGE),  troubleshooting,  repair,  verification,  and  teardown,  as 
well  as  for  maintenance  actions  of  the  AGE.  The  MTTR  was  then 
computed  for  CND  maintenance  actions  which  resulted  in  a  repair. 
The  latter  were  broken  down  by  troubleshooting  method:  automated, 
semi -automated ,  and  manual.  From  Fig.  E-15,  it  is  apparent  that 
the  variation  in  MTTR  for  the  different  modes  is  strictly  a  func¬ 
tion  of  the  troubleshooting  time  which  ranges  from  1.3  hours 
automatic  to  2.9  hours  manual.  There  are  obvious  benefits  for  end 


item  availability  and  manpower  consumption  when  the  BIT  does  the 
fault  isolation.  In  this  particular  test,  24  percent  of  the  324 
detections  were  CND.  Of  the  remaining  76  percent  where  corrective 
action  was  required,  51  percent  had  to  be  manually  isolated, 
resulting  in  the  longer  repair  times.  From  the  standpoint  of 
improving  availability  it  would  be  advantageous  to  maximize  the 
number  of  failures  isolated  by  BIT  and  reduce  the  manual  trouble¬ 
shooting  requirements. 

The  F/16  FOT&E  analyzed  measured  repair  time  for  automated 
and  manual  troubleshooting  for  two  categories:  flight  controls, 
and  multiplex  bus  avionics.  For  flight  controls,  the  BIT  MTTR  was 
3.63  hours;  the  manual  MTTR  was  significantly  longer:  11.07  hours. 

If  the  BIT  had  been  ineffective,  it  would  have  driven  a  very 
long  overall  MTTR.  However,  it  was  effective  for  92  percent  of  the 
maintenance  actions;  only  8  percent  were  manual.  The  result  was 
a  cumulative  MTTR  of  4.26  hours. 

For  the  multiplex  bus  avionics,  the  spread  between  BIT  and 
manual  MTTRs  was  not  so  dramatic:  2.05  hours  BIT  versus  2.71 
hours  manual.  The  BIT  was  effective  for  84  percent  of  the  main¬ 
tenance  actions  for  a  cumulative  MTTR  of  2.16  hours. 


c.  General  Conclusions. 


(a) 


BIT  is  necessary  for  fault  detection  and  isolation, 
particularly  in  complex  electronic  systems. 
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b.  In  general,  when  BIT  is  effective  for  fault  isolation, 
it  has  the  potential  to  significantly  reduce  repair 
times. 

c.  When  BIT  fault  detection  errs  in  the  direction  of 


high  erroneous  detections  it  can  adversely  impact 
mission  performance  by  affecting  crew  workload. 


It  also  contributes  to  downtime  and  maintenance  man¬ 
power  consumption.  On  a  1  for  1  basis,  CND  is  a 
lesser  contributor  to  downtime  than  a  failure  which 
requires  corrective  action. 

d.  CND  and  BCS/RTOK  rates  are  more  meaningful  when 
computed  on  a  flying  hour  (or  sortie)  basis  rather 
than  on  a  maintenance  action  basis  (though  the  latter 
is  also  a  useful  measure). 

e.  The  effect  of  high  CND  and  BCS/RTOK  rates  is  probably 
worse  in  periods  of  high  sortie  generation. 

f.  Once  a  system  is  fielded  and  tested,  it  is  difficult 
(to  impossible)  to  determine  real  BIT-caused  CND  and 
BCS  rates,  their  causes,  and  get-well  plans  without 
field  surveys.  It  is  important  to  recognize  that 
field  measures  can  vary  significantly  between  bases. 

g.  Once  a  system  is  fielded,  environmental  factors 
(human  judgment,  faulty  procedures,  maintenance 
policy,  training)  can  cause  high  CND  and  BCS/RTOK 
rates.  The  implication  for  design  is  that  BIT 
capabilities  should  be  so  thorough,  straightforward 
and  unambiguous  that  they  minimize  the  opportunity 
for  these  factors  to  cause  CNDs  and  BCSs/RTOKs. 
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h.  Minimizing  BCSs  is  important  for  a  three  level-main¬ 
tenance  concept  but  it  is  an  absolute  necessity  for 
a  two-level  concept. 

i.  A  significant  analytical  effort  is  required  in  the 
design  stage  to  optimize  BIT  (FD/FI,  false  alarms/ 
CND/BCS/RTOK ,  MTTR)  and  the  support  structure  (support 
equipment,  manpower,  skill,  spares,  technical  data. 
Better  techniques  are  required. 

C.  RECOMMENDATIONS  FOR  IMPROVING  DIAGNOSTICS 


During  the  study  effort,  many  recommendations  were  presented 
for  improving  diagnostics.  These  recommendations  address  program 
structuring,  technical  and  management  activities,  and  technological 
opportunities.  Recommendations  from  the  first  two  categories  are 
presented  here  in  the  same  basic  format  in  which  Current  Practice 
(para.  B)  was  addressed:  requirements,  design,  development  and 
demonstration  tests,  and  operational  test/maturation. 

From  the  case  study  activity  and  the  numerous  other  studies 
reviewed  and  presentations  made,  it  is  clear  that  the  achievement 
of  mature  diagnostics  capability  is  the  result  of  a  process. 

This  process  encompasses  both  research  and  development  activities 
which  are  not  weapon  program  specific,  as  well  as  the  entire 
acquisition  cycle  of  specific  weapon  programs.  Figure  E-16 
illustrates  the  context  in  which  diagnostic  considerations  must 
be  addressed. 

Generic  R&D  activities  have  focused  historically  on 
developing  performance-oriented  capabilities  to  counter  future 
threats.  Over  the  years,  with  the  realization  that  performance 
over  time  was  an  important  dimension  of  countering  threats, 
reliability  considerations  started  to  be  introduced  in  this 
area.  Only  in  the  last  several  years  diagnostics  capabilities 
have  begun  to  be  considered  as  important  activities  in  the  world 
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Non-weapon  Specific 


R&D  Activities 


-  Defense  Laboratories 


-  Contractor  IR&D 


-  Technology  Opportunities 


Weapon  System  Program 


-  Requirements 


-  Alternative  Concepts 


-  Demonstration/Validation 


-  Full  Scale  Development 


-  Production/Deployment 


FIGURE  E-16.  Diagnostics  Context 
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of  non-weapon-specific  R&D.  It  is  important  for  DoD  contractors 
to  continue  this  trend  so  that  technological  solutions  to  diag¬ 
nostic  problems  can  evolve  and,  more  importantly,  so  that  tech¬ 
nology  developments  focused  on  future  threats  consider  how  the 
eventual  systems  will  be  maintained. 

There  are  a  number  of  examples  where  diagnostics  are  being 
considered.  The  Integrated  Communications/Navigation/Identification 
Architecture  ( ICNIA)  and  Pave  Pillar  programs  are  Air  Force  programs 
oriented  toward  providing  advanced  architectures  for  future  avionics 
They  both  have  specific  objectives  to  address  diagnostics.  The  DoD 
Very  High-Speed  Integrated  Circuit  (VHSIC)  program  also  has  objec¬ 
tives  to  produce  reliable  and  testable  circuits.  Likewise,  several 
industry  IR&D  projects  are  developing  advanced  architectures  and 
equipment  which  are  formulated  with  reliability  and  diagnostics  as 
principle  characteristics. 

The  DoD  needs  to  continue  to  increase  this  emphasis  in  both 
government  laboratories  and  contractor  independent  R&D  efforts. 

Other  effects  should  be  considered  which  focus  on  developing  design 
tools  such  as  computer-aided  engineering  techniques  which  enable 
the  design  of  testable  circuits  and  systems. 

Referring  back  to  Fig.  E-16,  military  needs  have  evolved 
in  recent  years  to  considering  the  readiness  and  sustainability 
characteristics  of  force  structures.  All  the  services  have 
developed  policies  to  guide  the  development  of  future  forces  to 
meet  the  year-2000  operational  requirements.  These  projections 
have  in  common  the  need  for  improved  reliability  and  maintainability 
reduced  support  tails,  and  better  logistics  to  manage  more 
mobile  and  autonomous  units. 

Technology  R&D  and  military  needs  are  providing  opportunities 
and  requirements  for  improved  diagnostics  as  inputs  for  specific 
weapon  system  proyrams.  From  the  case  studies,  it  is  apparent 
that  diagnostic  capabilities  can  best  be  achieved  by  focusing 
on  diagnostics  as  a  fundamental  system  characteristic. 
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Achieving  effective  diagnostics  requires  a  plan,  management 
strategy,  motivation,  technical  activity,  and  funding  which 
spans  the  acquisition  activity  from  initial  requirements 
definition  through  deployment. 


Requirements 


The  military  user's  requirements  should  address  diagnostic 
capability  in  the  larger  context  of  the  operational  mission  and 
environment  as  well  as  the  support  constraints  of  manpower, 
skill,  maintenance  concept,  deployability  and  logistics  burden. 

The  requirements,  constraints,  environment  and  economics  should 
then  drive  the  architecture  of  the  system  with  diagnostics  as  a 
fundamental  characteristic.  Significant  improvements  are 
required  in  formulating  these  requirements. 

There  are  a  number  of  innovative  Service  efforts  to  define 
requirements  and  development  objectives  for  readiness  and  support 
at  the  front  end  of  a  weapon  program.  Examples  include  the  Air 
Force  Advanced  Tactical  Fighter  ( ATF) ,  The  Navy  Submarine  Advanced 
Combat  System  (SUBACS),  The  Army  Mobile  Protected  Gun  System  (MPGS), 
and  The  joint  Services  Advanced  Vertical  Lift  Aircraft  (JVX). 

Figure  E-17  summarizes  the  kind  of  performance-driven  support 
requirements  and  constraints  which  define  the  context  for  generating 
meaningful  diagnostic  requirements.  Objectives  which  call  for  re¬ 
duced  levels  of  maintenance,  high  utilization  rates,  self-sustain¬ 
ing  operations  and  reduced  support  tail  should  drive  the  develop¬ 
ment  of  high  confidence  bui lt-in-test  with  CND/BCS/RTOK  rates  of 
near  zero. 

Another  fundamental  issue  has  confounded  diagnostics  develop¬ 
ment  throughout  the  process;  namely,  definitions.  Definitions, 
terminology  and  figures  of  merit  to  describe  diagnostics  have 
proliferated  to  the  point  that  communication  relative  to  diagnostics 
measure  is  difficult.  This  is  not  a  trivial  problem;  it  impedes 
the  way  that  diagnostics  are  specified,  managed,  designed,  tested 
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ADVANCED  TACTICAL  FIGHTER 
( ATF) 


SUBMARINE  ADVANCED  COMBAT 
SYSTEM  (SUBACS) 


MOBILE  PROTECTED  GUN  SYSTEM 
(MPGS) 


JOINT  SERVICES  ADVANCED 
VERTICAL  LIFT  AIRCRAFT 
(JVX) 


•  ACHIEVE  90%  MISSION  RELIABILITY 
AT  HIGH  SORTIE  RATE  (X  SORTIES 

IN  6  DAYS,  Y  SORTIES  OVER  30  DAYS). 

•  DEPLOY  THE  SUPPORT  FOR  A  UNIT  IN 
6-7  C-141's. 

•  SMALL  UNIT  DECENTRALIZED  OPERATIONS. 

•  OPERATE  FOR  60  DAYS  OF  COMBAT  WITHOUT 
MAINTENANCE  AND  REMAIN  MISSION  CAPA¬ 
BLE. 

•  ACHIEVE  HIGH  RELIABILITY  AND  MAIN¬ 
TENANCE  SIMPLICITY  TO  OPERATE  ( INI¬ 
TIALLY)  WITH  EXTREMELY  LIMITED  LOG 
SUPPORT.  (TERMS  ARE  BEING  DEFINED). 

•  SUSTAIN  0.90  A  RATE  IN  WARTIME  FOR 
60-90  DAYS  FROM  UNIMPROVED  SITE. 


•  DETERMINE  INCREMENTAL  DESIGN  AND 
SUPPORT  REQUIREMENTS  TO  SELF  DEPLOY 
AND  OPERATE  AT  HIGH  SORTIE  RATES 
OVER  EXTENDED  PERIODS. 


•  SELF-CONTAINED,  X  ACCOMPANYING  A/C 

•  X , Y, Z  SORTIES  OVER  5  TO  90  DAYS 

FIGURE  E-17.  Defining  Development  Objectives  for  Readiness 
and  Support  at  the  Front-End 
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and  measured.  Proposed  MIL-STD-XXX,  "Testability  Program  for 
Electronic  Systems  and  Equipment",  and  MIL-STD-1309  are  useful 
but  not  sufficient  steps  to  resolve  this  problem. 

Better  ways  of  specifying  are  needed  to  achieve  the  readiness 
and  support  goals  of  the  Services.  One  proposal  was  presented 
which  appears  to  have  the  capability  of  influencing  reliability, 
maintainability,  support  costs,  and  readiness  to  achieve  a  two-level 
maintenance  concept.  The  approach  is  to  specify  that  all  avionics 
line-replaceable  units  meet  a  threshold  of  5XX.XX  acquisition  cost 
per  removal-free  operating  hour.  This  parameter  has  the  advantages 
of  being  operationally  useful  and  measurable  in  the  field  but  it  may 
not  communicate  clearly  to  the  designer. 
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2.  Design 


The  following  are  needed: 

a.  Strategies  to  minimize  CND,  BCS,  RTOK  and  false-alarm 
conditions  during  design. 

b.  Techniques  to  maximize  vertical  testability. 

c.  A  flexible  diagnostic  system  so  that  changes  can  be 
incorporated  readily  in  diagnostic  algorithms, 
screens  and  tolerances  with  minimal  hardware  impact. 

d.  Fault-free  software  development  techniques. 

e.  Techniques  to  enable  more  concurrent  hardware  and 
software  development  and  earlier  integration  of  the 
two. 

f.  Trade-off  tools  for  assessing  the  diagnostics  impli¬ 
cation  of  design  decisions  on  the  support  structure. 

g.  Computer-aided  engineering  techniques  for  enhancing 
design  for  testability  in  support  of  proposed  MIL-STD- 
XXX.  Some  techniques  such  as  Log Mod  and  STAMP  may 
already  be  able  to  meet  this  need  though  they  are  not 
used  widely. 

h.  Diagnostic  testing  in  the  early  operational  environ¬ 
ment  will  reveal  BIT  problems.  Recognize  that  matura¬ 
tion  phase  is  a  legitimate  design  activity  and  plan  for 
it. 

i.  Both  the  Services  and  contractors  need  to  cultivate 
experienced  groups  of  individuals  who  understand  and 
implement  good  diagnostics  designs. 

j.  Tools  for  predicting,  measuring,  and  managing  the 
diagnostics  development. 

k.  Better  design  practices  such  as  timing  margins  in 
high-speed  circuits  and  systems. 
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Development  and  Demonstration  Test 


a.  Use  reliability  and  other  test  events  as  opportunities 
to  discover  problems  with  BIT  performance.  Environ¬ 
mental  testing  may  be  particularly  useful  for  dis¬ 
covering  false-alarm  indications  such  as  induced  in- 
termittents  and  transients. 

b.  Consider  increasing  the  number  of  spare  assets  and 
budgeted  time  in  the  system  integration  laboratory 
so  as  to  be  able  to  investigate  diagnostic  anomalies 
without  impacting  the  schedule  and  use  of  other  assets. 

c.  Consideration  should  be  given  to  expanding  the  set  of 
faults  inserted  to  a  larger  set.  Time  required  and 
test  assets  might  have  to  increase. 

d.  Consideration  should  be  given  to  increasing  the  allow¬ 
able  cost  of  demonstrations  to  include  repair  costs, 
thereby  permitting  insertion  of  a  better  cross-section 
of  faults. 

e.  It  has  been  suggested  that  computer  simulation  of  the 
actual  circuits  can  test  much  of  the  BIT  provisions 
(hardware,  software,  firmware)  more  throughly  and 
without  causing  damage.  Consideration  should  also  be 
given  to  building  a  library  of  computer  simulation 
models  to  address  the  circuits  available  today  and  to 
keep  pace  with  evolving  technology. 

f.  Comparability  analysis  may  be  useful  as  a  method  of 
identifying  a  realistic  fault  set  for  insertion. 

g.  Improved  demonstration  techniques  which  better  predict 
field  diagnostics  performance  should  be  developed  and 
incorporated  in  MIL-STD-471. 


i 

* 

i 

i 

j  4.  Operational  Teat  and  Field  Maturation 

|  Field  maturation  is  essential  to  achieve  inherent  diagnostics 

potential.  The  techniques  and  activities  used  to  specify,  design, 
and  verify  diagnostics  capability  do  not  in  themselves  result  in  a 
mature  fielded  system.  When  the  system  is  first  fielded,  it  is 
common  to  find  that  not  all  the  hardware  and  software  provisions 
of  the  diagnostics  have  been  fully  implemented.  In  addition,  the 
operational  use  patterns  and  environment  produce  new  failure  modes 
and  diagnostic  indications.  These  new  indications,  which  the  BIT 
may  not  deal  with  properly,  are  resolved  by  the  judgment  of  opera¬ 
tors  and  maintainers  (who  may  not  have  been  trained  to  deal  with 
them)  and  with  the  aid  of  technical  data  (which  may  not  have  been 
developed  to  address  them).  A  structured  engineering  maturation 
effort  in  this  environment  is  the  only  way  to  bring  the  diagnostic 
capability  to  its  full  potential.  The  APG-65  and  APG-66  programs 
are  excellent  examples  of  effective  BIT  maturation.  The  key 
features  of  these  programs  should  be  used  in  structuring  future 
maturation  efforts  for  complex  equipment. 

The  key  features  of  diagnostics  maturation  are  as  follows: 

Planni ng »  The  government  program  office  along  with  the  prime 
and  key  subcontractors,  the  user,  and  the  operational  test  agency, 
must  recognize  that  complex  systems  require  diagnostics  maturation 
in  the  operational  environment.  This  recognition  must  be  coupled 
with  commitment,  funding  and  a  management  plan  to  pursue  diagnostics 
maturation  until  a  mature  capability  is  clearly  demonstrated  in  the 
intended  operational  environment.  The  schedule  for  this  effort  is 
dependent  on  many  factors.  The  APG-66  took  20  months  to  grow  to 
full  test  implementation  and  was  followed  by  another  24  months  of 
maturation.  The  APG-65  schedule  spans  34  months. 

Data  Collection:  A  special  diagnostics  data  collection  and 
analysis  system  is  required  to  capture  information  on  occurrences 
and  causes  in  enough  detail  to  provide  a  credible  data  base  for 
developing  and  implementing  engineering  solutions.  Navy  3M  and 
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Air  Force  MDC  are  not  sufficient  but  can  be  useful  for  this  pur¬ 
pose.  Other  inputs  include  the  specific  indications  and  circum¬ 
stances  logged  by  the  aircrew/operator,  a  BIT  debriefing  of  all 
missions,  the  specific  indications  of  detection  and  methods  of 
isolation,  a  specific  serial  number  track  of  LRUs/WRAs  through 
the  ultimate  repair  action  and  subsequent  performance  after  repair, 
as  well  as  the  elapsed  time  for  each  element  of  the  maintenance 
event  (set-up,  troubleshoot,  repair,  verification,  teardown ) . 
Analysis  can  then  focus  on  the  causes  of  alarms,  CNDs,  BCSs, 
no  detects  and  lengthy  repair  times. 

Recorders:  If  the  system  does  not  have  built  in  capability 

to  capture  the  detailed  environment  and  condition  information  at 
the  time  of  failure  indication,  additional  sensors  and  recorders 
should  be  installed  on  the  system  during  the  maturation  phase  to 
provide  this  information. 

Engineering  Manpower:  Knowledgeable  design  engineering  person 
nel  are  essential  at  operational  locations  to  observe  and  analyze 
the  performance  of  diagnostics  capabili.-.y  so  that  problems  can  be 
recognized  quickly  in  the  context  of  all  the  operational  environ¬ 
mental  variables. 

Maintenance  Manpower:  A  team  of  operational  maintenance 
personnel  (user,  supporter,  tester)  must  be  available  and  moti¬ 
vated  toward  the  objectives  of  maturing  the  diagnostics  and  in¬ 
stitutionalizing  preferred  troubleshooting  procedures  and  main¬ 
tenance  policy. 

Operational  Support:  The  operational  entity  which  is  employ¬ 
ing  the  new  system  must  be  charged  specifically  with  supporting 
diagnostics  maturation  and  must  not  be  so  overtaxed  with  train¬ 
ing  commitments  that  it  cannot  dedicate  sufficient  resources  to 
support  maturation  activities.  When  a  system  fails  or  exhibits 
a  diagnostic  problem  of  interest  to  the  maturation  team,  the 
emphasis  should  be  on  fully  exploiting  and  understanding  the  cause 
of  the  problem  rather  than  hurrying  the  system  back  into  commission 
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to  meet  a  mission-capable  rate  or  training  commitment.  These 
operational  units  must  have  sufficient  assets  assigned  to  meet 
both  operational  and  maturation  requirements. 

Data  Base  Resources;  Sufficient  computer  resources  (time, 
access,  programming)  must  be  available  at  the  operational  location 
to  maintain  a  diagnostics  maturation  data  base  and  to  support  timely 
analysis  by  the  maturation  team. 

Software  Discipline:  Since  many  BIT  anomalies  are  corrected 
by  software  changes,  a  vigorous  software  data  collection  and 
tracking  effort  is  required  to  update  and  control  the  software 
co:;f iguration.  Though  this  activity  is  normally  conducted  at 
the  contractor's  facility,  it  must  interface  closely  with  the 
field  maturation  effort.  The  APG-66  went  through  at  least  eight 
block  configuration  changes  and  one  major  ECP  during  maturation. 

The  APG-65  had  five  tape  configurations  in  the  first  20  months 
of  maturation. 

Contractor  Support  Base:  Contractor  resources  (time,  engi¬ 
neering  manpower,  and  system  integration  laboratory/software 
support  facilities)  are  required  as  a  support  base  for  the  field 
engineering  activity.  These  resources  are  necessary  for  verifying 
field  anomalies  as  well  as  formulating,  testing  and  implementing 
diagnostic  corrective  actions.  Production  testing  activities  at 
the  factory  may  present  other  opportunities  for  observing  addi¬ 
tional  failure  modes  and  diagnostic  indications.  These  opportun- 
ites  for  diagnostic  improvements  should  be  used  to  supplement 
the  field  maturation  activity. 

Contractor  Use  of  Diagnostics:  The  contractor  must  be 
required  to  use  the  BIT,  diagnostic  procedures,  and  technical 
data  being  developed  for  Service  use  whenever  he  is  performing 
maintenance;  for  example,  during  early  flight  testing  and  interim 
contractor  support.  These  events  can  present  early  opportunities 
for  maturation,  even  before  operational  testing  starts. 
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5.  An  Approach  to  Planning  Future  Avionic  Diagnostics 


The  following  section  is  presented  to  illustrate  how  a  system 
might  be  structured  at  the  front  end  to  achieve  significant  diag¬ 
nostic  improvement.  It  is  only  one  possible  approach  to  the 
solution.  It  is  specifically  oriented  toward  avionics  but  the 
thought  process  should  be  useful  for  other  applications  as  well. 
Regardless  of  the  system  type,  diagnostic  capability  must  be 
considered  as  a  fundamental  concern  in  the  conceptual  system 
archi tecture. 

In  the  world  of  avionics  diagnostics,  bold  steps  are 
necessary  to  improve  performance  radically  in  the  field  and 
substantially  reduce  the  cost  of  maintenance.  Supportability 
improvements,  particularly  the  contribution  of  avionics 
diagnostics,  must  take  on  a  new  approach  to  solve  the  problems 
faced  by  the  Services  in  the  field  today.  Technology  improvements 
clearly  offer  the  opportunity  to  make  strides  toward  such 
improvement.  Advanced  architectures  provide  the  means  to 
achieve  improved  supportability. 

At  the  outset,  system  improvement  objectives  should  be 
established  to  significantly  improve  supportability.  The  context 
for  these  objectives  is  formed  by  improved  requirements  definition 
as  discussed  in  paragraph  C.l  above.  Some  more  specific  goals  are 
listed  belowt 


SYSTEM  IMPROVEMENT  GOALS 

e  Malfunctions  which  Cannot  be  Duplicated  (CND) 
by  maintenance  personnel  not  to  exceed  X  per 
100  flight  hours. 

e  Bench-Checked  Serviceable  (BCS)  or  Retest  OK  ( RTOK ) 
events  not  to  exceed  Y  per  100  flight  hours. 
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•  Maintenance  to  be  accomplished  by  personnel  having  no 
higher  skills  than  the  skills  of  today's  personnel. 

•  Capability  of  autonomous,  self-sufficient  combat  opera¬ 
tion  at  dispersed  sites  with  no  avionics  intermediate 
shop  (AIS). 

•  Built-in  diagnostics,  health-monitoring  and  fault- 
management  functions  with  advanced  capabilities. 

One  method  which  has  been  suggested  to  achieve  these  and 
other  goals  is  documented  in  the  General  Dynamics  Advanced 
System  Integration  Demonstration  report  which  was  developed 
under  the  Pave  Pillar  program. 

This  concept  takes  a  new  approach  to  what  a  Line-Replaceable 
Unit  (LRU)  is  for  removal  and  replacement  maintenance  actions 
at  the  aircraft  level.  Smaller,  less-expensive  LRUs  are  defined 
for  use  in  advanced  packaging  concepts.  Each  LRU  is  self- 
sufficient  with  regard  to  diagnostics.  The  concept  is  depicted 
in  Fig.  E-18. 

This  approach  requires  the  application  of  new  technologies  and 
could  ultimately  result  in  significant  cost  reductions  in  support- 
ability  of  weapon  systems.  The  required  technologies  and  concepts 
are  listed  in  Fig.  E-19. 
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•  AVAILABILITY  •  SUPPORTABILITY  •AFFORDABILITY  •PERFORMANCE 


j  MAJOR  FEATURES}— 

•  SMALL  FAMILY  OF  STANOARO; 
PROGRAMMABLE  MODULES 
(VLSI/VHSIC  BiMd) 
v  Advanctd,  All-Mult  iplax 
Sytltmi  Architsclurt 


•  STANDARDIZATION  OF 
MOOULES,  RACKS  AND 


•  SELF  TEST  TO  AND 
REPLACE  MOOULES 
AT  AIRPLANE 


ASAP  ENHANCEMENTS 


•  ORDER  OF  MAGNITUDE  GAINS 


•  Operational  Reliability 

•  Reduce  CND&  RTOK  Rates 

•  Improve  Standardization 

•  Reduce  Maintenance  Time 

•  Reduce  Life  Cycle  Cost 


•  Reduce  Wiring  &  Connections 

•  Reduce  Hi-Value  Spare  Types 

•  Reduce  Spare  Types 

•  Hi-Spead  Data  Handling 

•  Reduce  Avionics  Shop 


•  MAJOR  GAINS 

•  Reduce  Skill  Requirements 

•  Replace  Modules  at  A/C 

•  Reduce  Initial  Avionic  Cost 

•  Increased  Performance 

•  Growth  Flexibility 


FIGURE  E-18.  Advanced  Avionics  Architecture 
for  Major  Improvements 

•  VHSIC  technology 

•  Modularity/module  level  self-test 

•  Two-level  maintenance/line  replacement 

•  Fault  tolerant  architecture 

•  On-line  spares 

•  Self  diagnostics 

•  Expert  systems  diagnostics 

FIGURE  E-19.  Technology  and  Concepts  for 
Reli abi li ty/Mai ntai nabi li ty/ 
Supportabi li ty  Improvements 
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A  key  to  this  new  architecture  is  the  recognition  that 
many  functions  performed  by  various  avionic  subsystems  are 
indeed  common,  particularly  in  digital  functions.  Figure  E-20 
illustrates  how  avionics  subsystems  can  be  partitioned  into 
common  functions. 
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FIGURE  E-20, 


Avionic  Subsystems  Can  be  Partitioned 
into  Common  Functions 
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Figure  E-21  shows  common  module  design  approaches  to  the 
different  types  of  common  functions  performed  within  avionic 
system  elements. 

The  key  features  of  this  approach  are: 

•  Groups  of  common  modules  constitute  traditional 
subsystems 

•  Communication  within  subsystems  over  high-speed 
parallel  bus 

•  Communication  within  subsystems  over  serial  bus 

•  Extensive  self-test  capabilities 

••  Test  processor  in  each  cluster 
••  System-level  maintenance  processor 

•  Integrated  airplane  racks 

•  Directly  usable  in  airplane  and  ground  systems. 


•  BUILDING  BLOCK  APPROACH 
•STANOARD PARTS 

•  NEW  TECHNOLOGY  BASEO 


IASKCOAMON  ELEMENT 


MHI  CAM 
MKAOCOMFDTER 


MASS  MEMORY 
ELEMENTS 


immi 

FROCAAM 

MEMAV 


.  STANOARD  SACK -RACK  MODUS  WITH  OMfCT 
ISOIBS'-OrilCAl  INTERFACE  TO  MtCROCOUTUTEA 


5/^ 


-^1 

ANAVOS  TO 
DIGITAL 
CONVERSION 
ELEMENTS 


vy 

OIGITAl  TO  OICITAl  TO  VECTOR 

ARA106  OISCRITIOR  FAOCESSOR 

CONVERSION  OISCRITE  TO  MOOVLIS 

ELEMENTS  DIGITAL  CONVERSION 
ELEMENTS 


KKSVlT  HIGHLY  FLEXIOLE  FAMILY  OF  STANOARO  OIGITAL  MODULES 
•  RN0CN4MK4A It  .  Alt  ttUM  I*  ft At  ACt  •lOttCOtT 


Ml  ONITCNMO 
CENTER 


FIGURE  E-21. 
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Common  Functions  Can  Be  Performed 
By  Standard  Modules 


The  method  suggested  to  achieve  a  two-level  maintenance 
concept  depends  on  sel£-test  that  is  self-contained  and  compre¬ 
hensive  within  individual  modules.  Common  module  design  with 
these  attributes  is  achievable  through  VHSIC  and  advanced 
architectures.  Figure  E-22  shows  how  this  approach  uses  an 
integrated  rack  for  the  modules  and  lists  the  expected  benefits 


SELF-TEST  TO  AMO 
REPLACE  MOOUIES 


support  iowpment 


•  ELIMINATE  MOST  AVIONICS  INTERMEDIATE  SHOP 

•  A  IS  EQUIPMENT.  TO  j  I  SPARES 

•  A  IS  PERSONNEL  S  TRAMMS 

•  Alt  LOGISTICS  SUPPORT 

.  AIS  EQUIPMENT  SOFTWARE  S  SUPPORT 

•  REDUCE  CND.  RTOK.  0  REPEAT  PROOLEMS 

•  NOLATE  PROOLEMS  IH-F  LISNT  AS  TRET  OCCUR 

•  REPLACE  SINOLE  MODULES  AT  AIRCRAFT 

•  ELIMINATE  HIGH-OOLIAR  SPARE  LRIk 

•  REOUCE  MAINTENANCE  TIME  0  MAN-HOURS 

•  REOUCE  SKILL  REQUIREMENTS 

•  REDUCE  GROUND  TRANSPORT  PROOLEMS 


FIGURE  E-22.  Automatic  Self -Test  to  Single  Nodule 
Will  Revolutionize  pupport 
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A  summary  of  this  approach  and  its  potential  gains  in  sup- 
portability  are  shown  in  Fig.  E-24. 


wide  use.  vsu 

SASEOSTANOARO 

MODUlfJ 


SUPPORT  ABILITY 
IMPROVEMENTS 


AOVANCEO. 
All  MULTIPLEX 
SYSTEMS 
ARCHITECTURE 


FAULT 

TOLERANT 

ARCHITECTURE 


seif test 

IN  F LIGHT 


integrated 

AVIONICS  RACKS 


I  TO  SINGH  MOOUli  %  SEAlEO  MOOULES 


OUAl/MUl 
COMMUNICATIONS 
I  INN 


INCREASED  OPERATIONAL  RELIABILITY 


REDUCED CNO &  RTOK  RATES 


REDUCED  NO  Of  SPARE  TYPES 


REDUCED  WIRING  A  CONNECTORS 


REOUCEO  HI  VALUE  SPARES  TYPES 


REOUCEO  MAINTENANCE  man  hours 


REOUCEO/ELIM  INTERMEDIATE  SHOP 


REOUCEO  SKILL  REQUIREMENTS 


REPLACE  MOOULES  AT  A/C 


REOUCEO  MODULE  COST  -  HIGH  PRODUCTION 


FIGURE  E-24.  Summary  Overview  of  New  Avionics  Approach 
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The  potential  payoffs  of  this  approach  can  be  substantial 
in  the  areas  of  reduced  cost  for  spares  and  maintenance  hours 
and  at  the  same  time  provide  higher  system  availability. 

•  Reduced  system  cost 


•• 

Fewer  module  type 

•• 

Fewer  spare  types 

•• 

Larger  production 

•• 

Less  maintenance 

quanti ties 

personnel 

•• 

Less  support  equipment 

•• 

More  commonality 

Higher  system  availability 

•• 

Increased  reliability 

•• 

Improved  off-site 
deployment 

•• 

Quicker  turnaround 

•• 

Affordable  spare 

level 

A  systematic  appi  jach  to  new  technology  programs  is  required 
to  improve  diagnostics  and  fault  management,  achieve  reductions 
in  support  cost,  and  improve  the  readiness  and  sustainability 
of  weapon  systems.  A  recommended  approach  is  to: 

« 

•  Establish  an  umbrella  DoD  laboratory  program 
for  diagnostics. 

•  Establish  R&M  programs  to  reduce  unnecessary 
maintenance  to  near  zero  for  current  and 
future  weapon  programs. 

•  Develop  special  diagnostic  hardware;  for  example, 
non-volatile  memories  for  "soft"  failures. 

•  Focus  on  future  performance-driven  support 
requi rements . 
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•  Develop  industry  standards  for  design  practice; 
for  example,  design  margins  for  timing  functions 

•  Demonstrate  diagnostic  R&D  projects  in  an 
operational  environment. 

•  Collect  and  analyze  field  data  and  current 
systems  to  better  understand  problem  causes. 

•  Include  diagnostics  as  a  basic  consideration 
in  R&D  programs  which  are  oriented  to  advanced 
system  architectures,  hardware  and  software. 
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A.  AFFORDABILITY 

During  past  years,  many  tasks  have  been  undertaken  to 
identify  the  cost  of  Reliability  and  Maintainability  for 
typical  weapon  system  programs.  (Sven  though  senior 
management  within  the  Department  of  Defense  remains  interested 
in  capturing  the  costs  associated  with  discrete  reliability 
and  maintainability  activities,  there  is  no  evidence  that  a 
work  breakdown  structure  (WBS)  change  has  been  made  to  require 
the  collection  of  cost  in  a  manner  that  would  support  this 
desire . 

Currently,  the  costs  of  reliability  and  maintainability 
activities  are  contained  in  varying  amounts  throughout  the 
typical  WBS  as  shown  in  Fig.  F-l  below  for  aircraft  systems. 

TYPICAL  AIRCRAFT  SYSTEM  WBS 
AIRCRAFT 


PECULIAR  | 
SUPPORT 
EQUIPME 


INING 


NDUS' 

’acil: 


UsrW 


Even  though  the  WBS  has  not  historically  provided  the 
necessary  detail  to  allow  the  dissecting  of  costs  associated 
with  R&M ,  the  desirability  of  doing  it  remains.  Throughout 
the  course  of  the  case  study  development  and  analysis,  efforts 
have  been  made  to  collect  the  costs  associated  with  particular 
R&M  activities.  In  a  limited  number  of  cases  cost  data  have 
been  collected.  Case  histories  document  cases  of  reduced 
life-cycle  costs  (LCC)  associated  with  the  performance  of  a 
specific  reliability  activity.  In  other  cases  it  was  possible 
to  identify  that  the  absence  of  certain  reliability  or  maintain¬ 
ability  activity  has  led  to  supportability  problems  once  the 
system  was  fielded. 

The  cost  of  a  typical  reliability  and  maintainability 
program  is  composed  of  both  fixed  and  variable  components. 

Even  though  decisions  to  do  more  or  less  of  a  specific  activity 


are  normally  based  on  the  marginal  cost  associated  with  a 
expected  benefit,  it  is  not  desirable  to  structure  an  entire 
R&M  program  based  on  anything  other  than  a  balanced  program. 
Tasks  performed  that  lead  to  a  more  reliable  or  maintainable 
system  are  strongly  interrelated.  For  example,  if  black-box 
level  screens  were  100  percent  effective,  lower-level  screens 
at  the  subassembly  and  part  level  could  be  reduced  without 
significant  impact  on  field  reliability.  However,  due  to  the 
differences  in  costs  of  repairing  failures  at  different  levels 
of  assembly,  this  approach  would  have  a  different  cost  for 
repair  than  a  coordinated  screening  program  which  screens  out 
failures  at  the  most  cost-effective  level,  i.e.,  part,  sub- 
assembly,  black  box  or  system.  With  that  in  mind,  we  have 


& 


begun  our  evaluation  of  the  affordability  of  R&M  with  a  look 
at  the  baseline  costs  associated  with  programs  of  differing 
sizes  and  complexities  to  define  the  resources  that  a  program 
might  consume  in  manpower-related  activities.  For  some 
typical  program  tasks  like  analysis,  the  total  resources 
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required  may  be  directly  related  to  manpower.  For  other 
tasks  such  as  testing,  the  cost  components  might  include 
manpower,  facilities  and  the  hardware  to  be  tested. 

Following  the  evaluation  of  the  baseline  costs  of  R&M , 
the  discrete  costs  of  select  activities  that  are  performed 
for  improving  reliability  and  maintainability  are  evaluated 
for  payback  associated  with  those  activities.  The  analysis 
looked  specifically  at  the  costs  of:  maturation  testing, 
focusing  primarily  on  the  APG-65,  the  APG-66  and  the  T700; 
typical  Environmental  Stress  Screening  (ESS)  programs;  and 
MIL  STD  versus  commercial  parts  in  terms  of  the  implications 
associated  with  part  quality  and  cost  and  the  maintenance 
costs  associated  with  CNDs  and  RTOKs . 

1.  Baseline  Costs  for  R&M 

The  objective  of  the  program  case  studies  has  been  to 
develop  a  credible  list  of  engineering,  design,  test  and  con¬ 
tracting  activities  which  if  followed  will:  improve  R&M 
levels,  be  predictable  in  cost  and  schedule,  be  discernible 
in  terms  of  effect  on  equipment  design,  be  relatable  to  pay¬ 
offs  in  reducing  support  cost,  improving  readiness,  and  be  effec 
tive  in  accelerated  acquisition  (concurrent)  programs.  Other 
sections  of  this  case  study  analysis  address  portions  of  this 
objective  while  this  section  establishes  a  baseline  for  R&M- 
related  manpower  costs  to  perform  specific  related  tasks. 

Apart  from  the  case  studies,  other  sources  were  examined 
in  order  to  better  understand  the  costs  associated  with  R&M. 
These  efforts  to  shed  some  light  on  probable  development  costs 
are  not  a  panacea,  but  should  help  consolidate  recent  findings 
on  the  total  manpower  costs  of  R&M  programs  as  well  as  providing 
a  means  of  estimating  the  manpower  portion  of  the  task  costs  of 
R&M  activities  during  the  development  phase  of  the  system  life 


F-5 
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cycle.  In  some  cases,  manpower  makes  up  most  of  the  cost 
associated  with  the  task  element. 

The  intent  here  is  not  to  develop  a  methodology  for  estima¬ 
ting  the  cost  of  reliability  in  a  program  but,  instead,  to  use 
data  collected  by  individual  case  studies  to  make  inference  as 
to  the  cost  of  selected  activities  that  generally  are  accepted 
as  contributing  to  improved  reliability  in  program  development. 
With  these  guidelines,  managers  at  all  levels  should  at  least  be 
able  to  make  rational  judgments  about  the  manpower  costs  asso¬ 
ciated  with  specific  task  activities  within  a  program.  These  data 
added  to  the  costs  of  special  facilities  and  hardware,  should 
provide  a  fair  estimate  of  the  costs  associated  with  the  tasks. 

The  allocation  and  accounting  for  costs  in  a  typical  pro¬ 
gram  are  accomplished  using  an  established  WBS,  the  accounting 
method  used  to  collect  cost  by  activity,  and  usually  will  not 
support  an  analysis  to  determine  the  cost  of  reliability  activi¬ 
ties  like  "Environmental  Stress  Screening"  and  R&M-related 
design  activities.  A  recent  study  by  Hall,  Milliren  and  Schneider 
of  the  Boeing  Aerospace  Company  identifies  the  manpower  costs  of 
reliability  activities  (see  Fig.  P-5)  in  13  missile  and  space 
programs.  Reliability  program  costs  were  presented  relative  to 
the  engineering  program  budget  measured  in  man-months  per  year 
(MM/YR)  as  shown  in  Fig.  F-2. 

Considering  only  the  major  programs,  the  range  of  engi¬ 
neering  efforts  for  one  year  in  the  major  program  is  shown  for 
programs  during  the  validation  and  FSD  phases  in  Fig.  F-3. 

The  authors  suggest  „hat  three  qualifiers  should  provide 
insight  to  the  users  for  making  estimates  from  the  above  table. 
First,  system  complexity  directly  affects  the  engineering 
program  total  activity — the  more  complex  the  system,  the 
higher  the  cost.  Secondly,  the  degree  of  concurrency  in  the 
program  is  a  major  influence;  more  concurrency  requires  higher 
annual  engineering  budgets.  Finally,  program  size  is  itself 
an  influence — generally,  the  higher  the  production,  the  higher 
the  cost. 
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Using  the  average  program  values  for  both  engineering  and 
reliability  program  (MM/YR)  expenditures,  the  size  of  the  relia¬ 
bility  program  can  be  estimated,  relative  to  the  total  engineering 
program.  During  FSD,  the  range  of  total  engineering  expenditures, 
relative  to  the  average  program  was  roughly  +30  percent  (see 
Fig.  F-3). 

Knowing  the  size  of  the  reliability  program  expenditure  in 
relation  to  the  total  engineering  budget  provides  insight  for  a 
major  portion  of  the  cost  associated  with  many  of  the  reliability 
tasks  that  might  be  a  part  of  a  development  program.  These  tasks 
are  listed  in  Fig.  F-5.  A  caution  worth  remembering  is  that 
these  expenditures  are  for  "people  cost  only"  to  some  specific 
reliability  tasks. 

Reliability  program  expenditures  ranged  from  a  high  of  427 
to  a  low  of  6.4  MM/YR  for  the  13  programs.  The  corresponding 
level  of  effort,  relative  to  the  engineering  program  MM/YR 
ranged  from  1.5  percent  to  13.1  percent.  Figure  F-4  portrays 
averages  of  these  factors  for  the  13  programs. 


PROGRAM  CATEGORIES 


FIGURE  F-4.  Reliability  Program  Expenditures 
Percent  of  Engineering  Budget 
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As  can  be  seen  from  Fig.  F-4,  the  average  expenditure  for 
R&M  tasks  listed  in  Fig.  F-5  was  approximately  four  percent.  In 
a  separate  study,  the  Naval  Sea  Systems  Command  places  the  cost 
of  the  average  reliability  program  at  4.1  percent  of  the  total 
contract  value  for  FSD.  This  finding  was  based  on  a  study  of  22 
complex  electronic  systems  built  by  17  contractors.  In  general, 
they  found  that  reliability  program  expenditures  decreased  to 
about  2.5  percent  as  the  size  of  the  program  increased.  This 
finding  is  similar  to  those  documented  in  the  Boeing  study.  It 
should  be  noted  that  in  both  the  Navy  and  Boeing  studies  the 
percentage  of  FSD  cost  for  R&M  does  not  include  the  effort  extended 
by  designers  toward  R&M. 

In  yet  another  attempt  to  identify  R&M  program  costs,  the 
Air  Force  Acquisition  Logistics  Division  of  AFLC  stated  that  the 
costs  of  R&M  are  not  readily  visible.  Those  that  are  visible 
are  generally  only  for  the  R&M  department  and  do  not  include 
the  effort  expended  by  designers  toward  R&M.  Great  care  must  be 
taken  if  R&M  costs  as  a  percentage  of  development  costs  are  to  be 
used  as  an  indicator  of  program  goodness.  Examples  of  this  are 
that  a  high  percentage  could  indicate  poor  system  integration 
of  R&M  while  a  lower  percentage  could  indicate  either  good 
integration  or  an  underestimation  of  the  effort  required. 

Manpower  cost  for  reliability  tasks  can  be  derived  from 
the  Boeing  study  mentioned  previously.  Costs  associated  with 
the  specific  tasks  are  shown  in  Fig.  F-5.  This  further  breakdown 
of  the  reliability  costs  by  task  can  be  useful  in  formulating 
estimates  for  specific  activities.  When  the  particular  task  is 
manpower-intensive,  the  cost  may  be  reasonably  close  to  these 
estimates.  For  other  activities  such  as  ESS  or  testing, 
the  cost  of  facilities  and  hardware  must  be  added  to  obtain  a 
representative  cost.  Figure  F-5  depicts  manpower  costs 
for  each  of  the  reliability  WBS  tasks  identified  plus  an  estimate 
of  other  Engineering  Support.  This  other  support  is  representative 
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of  the  tasks  that  design  engineers  provide  in  support  of  those  re¬ 
liability  and  maintainability  tasks  listed  in  Fig.  F-5. 


Program  Task  FSD  Phase 


Reliability  Program  Plan  .1 

Subcontractor  Control  3.6 

Program  Reviews  1.7 

FRACAS  and  FRB  10.5 

Reliability  Analysis  4.9 

FNECA  8 . 4 

Sneak  Circuit  Analysis 
Electronic  Parts/Circuit  Tolerance 
Analysis  3.1 

*  Parts  Program  51.0 

Critical  Items  .1 

Effects  of  Test,  Storage,  etc.  .7 

Reliability  Testing  9.8 

Other  Engineering  Support  6.0 


Total  100.0 

FIGURE  F-5.  Expenditures  for  Reliability  Tasks 
Percent  of  Reliability  Budget 

T  The  manpower  costs  associated  with  the  parts  program  appears 
higher  than  on  the  typical  electronic  program.  The  Naval  Sea 
Systems  Command  study  places  this  value  at  33  percent  for  the  22 
programs  it  evaluated.  The  Boeing  study  looked  at  only  missile 
and  space  programs  where  in  the  past  there  was  more  emphasis  on 
the  parts  control  program. 
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2.  Costs  and  Benefits  of  Program  Maturation 

Without  exception,  each  of  the  case  studies  has  identified 
testing  as  a  vital  part  of  all  R&M  programs.  Testing  identifies 
problems  that  can  be  solved  in  a  variety  of  ways  that  include 
design  changes,  manufacturing  changes  and  revised  operating  proce¬ 
dures.  The  idea  that  R&M  can  be  improved  through  an  overall  pro¬ 
gram  of  testing,  designed  to  mature  the  hardware,  is  at  issue  here. 
Bven  though  case  studies  did  not  identify  an  overall  "maturation 
program,"  the  testing  that  was  conducted  did  produce  maturing 
results  in  varying  degrees.  The  vital  elements  of  any  maturation 
program  can  be  reduced  to  orderly  testing  of  the  hardware,  analyzing 
the  failures  that  are  identified,  incorporating  fixes  and  then  measur¬ 
ing  results.  The  tests  used  to  perform  this  function  vary  from 
program  to  program,  but  in  the  final  analysis  each  of  our  case 
study  programs  has  tested  the  hardware,  analyzed  the  failures,  and 
taken  appropriate  corrective  action,  and  a  more  reliable  product 
resulted. 

Testing,  like  other  activities  within  a  development  program, 
costs  money.  Program  managers  are  constantly  faced  with  budgetary 
constraints  that  force  them  to  tailor  their  development  programs. 

The  intent  is  to  quantify  the  costs  and  benefits  associated  with 
several  testing  programs  and  to  provide  substantiation  for  includ¬ 
ing  maturation  testing  in  a  program  structure  because  of  the  high 
return  that  it  offers. 

In  the  case  of  the  APG-66  radar  the  benefit-to-cost  ratio  for 
maturation  tests  performed  was  approximately  20:1.  Additionally, 
payback  periods  are  quite  short,  occurring  at  the  230,000  flying- 
hour  point,  a  milestone  the  F-16  passed  in  April  1983.  There  also 
are  substantial  reasons  to  believe  that  benefits  achieved  to  date 
are  conservative  estimates  that  can  be  improved  still  further, 
given  adequate  incentives. 
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At  the  outset,  it  is  important  to  understand  that  the  APG-66 
did  not  have  a  "Maturation  Test  Program,"  per  se.  This  analysis 
evaluates  the  combined  effects  of  several  discrete  tests  and  uses 
the  output  to  obtain  results  that  demonstrate  an  improved  relia¬ 
bility.  For  this  analysis,  this  maturation  test  program  included 
both  growth  and  qualification  testing. 

Testing  began  in  mid-1977  with  the  FSD  growth  test  and  for 
purposes  of  this  analysis,  was  completed  with  the  Production  Relia¬ 
bility  Qualification  test  in  late  1980.  It  is  likely  that  the  full 
benefit  of  all  corrective  actions  taken  as  a  result  of  those  tests 
has  not  yet  been  fully  realized.  An  additional  420  hours  of  test¬ 
ing  was  conducted  subsequent  to  these  tests  and  47  corrective  actions 
were  taken  to  correct  identified  failures. 

For  this  analysis,  however,  benefits  stemming  from  corrective 
actions  must  have  been  achieved  and  measured  in  the  field  during  a 
period  of  time  roughly  equivalent  to  the  calendar  time  for  maturity 
testing.  For  example,  the  total  calendar  time  for  the  maturity 
testing  covered  approximately  42  months.  During  these  42  months, 
over  2800  test  hours  were  accumulated  on  the  radar,  failure  modes 
were  identified  and  corrective  actions  taken.  Specific  tests  and 
test  hours  are  shown  in  Fig.  F-6. 

The  benefits  that  accrued  as  a  result  of  these  tests  have 
been  measured  over  a  calendar  period  approximately  equal  to  the 
test  period  after  the  equipment  reached  the  field.  Field  relia¬ 
bility  values  are  as  reported  by  the  Air  Force  Maintenance  Data 
Collection  System  (AFM  66-1)  and  are  shown  in  Fig.  F-7 .  Note 
the  4;1  improvement  in  reliability  over  the  period.  Mean  Time 
Between  Maintenance  (MTBM )  Type  1,  is  the  surrogate  used  for 
operational  reliability. 

Costs  associated  with  the  maturity  testing  have  been  calcu¬ 
lated.  These  expenditures  represent  less  than  10%  of  the  total 
costs  of  the  APG-66  development  contract  and  will  result  in  a 
significant  cost  avoidance  over  the  life  of  the  system. 
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Test  Type 
FSD  Growth 
FSD  Pre  ROT 
FSD  ROT 
Prod.  Growth 
Prod.  ROT 


Test  Hours 
420 
470 
730 
230 
956 


Total 


2806 


Costs  for  maturity  testing  are  presented  in  Fig.  F-8. 
These  costs  do  not  include  the  baseline  manpower  related  costs 
of  reliability  presented  in  Section  A-l  previously. 


Total  Test  Hours 

2806 

Cost  per  test  hour 

$700 

Subtotal 

$2.0M 

Test  Asset  Cost 

.  6M 

Number  of  test  assets 

3 

Subtotal 

$1.8M 

Total  Maturity  Test  Cost 

$3.8M 

FIGURE  F-8.  APG-66  Radar  Maturity  Test  Costs 

Return  on  investment  (ROI)  for  the  maturity  testing  activity 
and  the  payback  period  are  indicators  of  the  value  of  activity. 

ROI  is  conservatively  20 si.  This  value  can  be  demonstrated  using 
the  field  reported  MTBM  (TYPE  1)  for  the  3rd  quarter  1979  (14 
hours),  the  4th  quarter  1982  (52  hours)  and  the  Operating  and 
Support  (O&S)  cost  avoidance  from  Fig.  F-9. 

This  cost  avoidance  is  a  result  of  all  the  factors  that 
influence  change  in  MTBM  TYPE  1.  These  factors  include  the  results 
of  maturity  testing,  RIW  contract,  the  improved  training  of  person¬ 
nel  that  results  from  having  the  equipment  to  work  on,  maturing 
of  the  logistics  activities  that  support  the  system  and  an  improved 
pilot  understanding  of  how  the  system  works.  Maturity  testing, 
a  major  contributor  to  this  change  is  estimated  to  account  for 
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b .  Cost  Benefit  of  the  TPQ-37  Reliability  Improvement  Program 

The  TPQ-37  offers  another  good  example  of  how  reliability 
improvements  provided  for  improved  readiness  of  fielded  hardware 
with  a  return  on  investment  (ROI)  estimated  to  be  15:1.  In 
this  case,  the  TPQ-37  was  somewhat  unique  in  that  it  progressed 
from  Advanced  Development  (AD)  into  production  without  a  full 
scale  development  phase.  Early  in  the  production  phase,  the 
Army  funded  an  ECP  as  a  part  of  a  reliability  improvement  pro¬ 
gram  (RIP)  aimed  at  improving  the  field  reliability. 

The  RIP  was  funded  at  S5.5  million.  Improvements  in  reli¬ 
ability  during  the  period  are  shown  in  Fig.  F-ll.  This  improv  ent 
from  33  hours  MTBF  to  more  than  120  hours  results  in  a  cost  s?  t;.gs 
of  more  than  $80  million,  based  on  reduced  repair  cost  for  th 
system  relative  to  currently-reported  system  failure  rates.  l 
savings  was  calculated  using  a  nominal  cost  of  $5000  per  field 
repair  action.  The  number  of  failure  events  during  the  estimated 
20-year  life  of  the  TPQ-37  was  reduced  by  over  16,000  as  a  result 
of  the  improved  reliability. 

The  payback  period  for  the  RIP  is  also  quite  short.  Even  with 
the  relatively  slow  production  rate,  the  payback  for  the  total 
$5.5  million  contract  is  estimated  to  occur  in  1985. 
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Cost  Aspects  of  Environmental  Stress  Screenin 


One  definition  of  ESS  is  "the  application  of  a  specific 
combination  of  environmental  stresses  on  an  accelerated  basis, 
but  within  design  capability,  in  an  attempt  to  surface  latent  or 
incipient  flaws  which  in  all  likelihood  would  show  up  in  the 
operational  environment"  (ESS  of  Electronic  Hardwares  "To  Know 
It  Is  to  Love  It",  Neil  Mandel).  Several  methods  have  been  used 
over  the  past  few  years  in  an  effort  to  improve  the  ESS  process. 

To  date,  evidence  suggests  that  the  specific  environments  and 
durations  must  be  tailored  to  the  hardware  to  be  screened  in 
order  to  be  effective,  and  there  is  no  universally  accepted  ESS 
technique  that  can  be  applied  to  all  types  of  parts  and  equipments 
Generally,  however,  it  is  agreed  that  varying  amounts  df  random 
vibration  and  thermal  cycling  do  precipitate  flaws  that  might 
show  up  later  in  the  operational  environment.  Removing  these 
flaws  early  in  the  manufacturing  process  translates  to  lower 
production  costs  and  improved  operational  reliability  and  main¬ 
tainability  performance.  It  is  important  to  note,  however, 
that  removing  flaws  is  not  enough.  A  systematic  appraisal  of 
why  the  part  failed  and  the  correction  necessary  to  preclude  its 
recurrence  is  mandatory  for  optimum  results. 

The  cost  benefits  associated  with  screening  bad  parts  at  the 
lowest  level  in  the  manufacturing  process  can  be  quantified. 

One  study  has  shown  that  defective  ICs  will  cost  $75  to  correct 
at  the  subassembly  level  and  as  much  as  $200  at  the  assembly 
level.  Part  failures  corrected  at  the  system  level  may  cost  in 
excess  of  $1000  and  once  in  the  field  will  probably  cost  over 
$2000.  To  be  economically  prudent,  the  cost  of  screening  should 
be  less  than  the  cost  of  repair  during  in-house  testing  of  the 
hardware  prior  to  shipment  to  the  customer  with  the  by-product 
of  improved  operational  performance.  Some  companies  may  find  the 
capital  expenditure  for  equipment  to  perform  the  necessary  screens 
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to  be  excessive.  Estimates  vary,  but  costs  to  implement  100  percent 
testing  of  components  at  three  temperature  levels  could  cost  in 
excess  of  $3M  for  the  capital  equipment  (Ref.  Nov.  16,  1982,  R&M 
Core  Group  Briefing  by  J.  L.  Capitano) 

Another  source  of  information  from  a  manufacturer  of  screen¬ 
ing  facilities  (Screening  Systems,  Inc.,  El  Toro,  CA)  indicates 
that  a  combined  environments  (temperature  cycling  and  tri-axial 
quasi-random  vibration)  chamber  with  a  30"-diameter  head  and  450- 
pound  capacity  can  be  purchased  for  about  $100K.  These  chambers 
are  currently  in  use  on  the  Navy  MK-48  Torpedo  program.  Additional 
information  on  the  facilities  cost  aspects  of  combined  environ¬ 
ments  test  facilities  is  contained  in  "Mission  Profile  Test  Costs, 
Facilities  and  Test  Performance,"  AFWAL  TR  80-3087,  September  1980, 
published  by  the  Air  Force  Flight  Dynamics  Laboratory,  WPAFB ,  OH. 

The  marginal  cost  of  performing  ESS  at  the  subassembly  and 
system  level  is  $20  and  $40,  respectively,  based  on  one  experience 
at  Gould  for  a  5000-part  system.  Considering  the  range  of  air¬ 
borne  radars  from  the  case  studies,  9500  parts  for  the  APG-66  and 
18,830  parts  for  the  APG-63,  and  assuming  100  parts  per  subassembly, 
one  might  infer  that  the  cost  of  screening  an  airborne  radar  would 
be  from  $1900  to  $3800  for  all  subassemblies  and  from  $760  to 
$1500  each  at  the  system  level.  The  total  cost  for  ESS  of  each 
system  would  range  from  about  $2660  to  $5300.  A  reference  set  of 
stress  screening  cost  models  is  included  in  the  Institute 
of  Environmental  Sciences'  publication  titled  "Environmental  Stress 
Screening  Guidelines." 

The  benefits  that  accompany  these  costs  could  prove  to  be  the 
most  important  aspect  of  the  relationship.  Results  of  ESS  (random 
vibration)  by  IBM  on  the  F-15  Central  Digital  Computer  indicate 
significant  increase  in  the  failure  rate  of  the  50  computers 
screened,  which  led  to  a  6-to-l  reduction  in  infant  mortality  observed 
during  prime  integration  test.  In  a  separate  test  on  the  F-15  radar 
subsystem,  a  six-day  reduction  in  the  manufacturing  schedule 
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was  realized  through  ESS.  As  a  result,  Hughes  Aircraft  reduced 
the  cost  of  each  radar  to  the  Air  Force  by  $800. 

The  evidence  seems  clear.  Environmental  stress  screening, 
properly  administered,  can  help  ensure  hardware  performance  and 
reduce  the  cost  of  rework  while  minimizing  schedule  delays  that 
accompany  the  rework  and,  most  importantly,  reduce  manufacturing 
defects  in  delivered  hardware.  The  resultant  improved  reliability 
will  reduce  system  operating  and  support  cost  throughout  its  useful 
life,  while  at  the  same  time  also  improving  the  operational  readi¬ 
ness  of  the  system. 
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4.  Depot  Repair  Cost  Savings  for  the  ARN-84 

Gould  NavCom  System  division  reported  their  experience  using 
Environmental  Stress  Screening  on  the  ARN-84  TACAN.  Their  ESS 
emphasized  the  use  of  thermal  shock  to  precipitate  failures  at 
the  piece-part  level.  Prior  to  implementation  of  this  process, 
the  Navy  was  reportedly  experiencing  approximately  200  equipment 
operating  hours  mean  time  between  failure  with  the  ARN-84.  After 
implementing  this  process,  the  Navy  is  reportedly  experiencing 
approximately  2000  equipment  operation  hours  mean  time  between 
failure.  Based  on  these  two  MTBF  values,  the  cost  savings  for 
only  the  depot  level  repair  for  the  ARN-84  was  calculated  to 
be  in  excess  of  $5  million  per  year  for  the  5000  sets  currently 
in  use  by  the  Navy.  The  cost  of  depot  level  repair  is  shown  in 
Fig.  F-12  below  for  these  two  MTBF  values. 


200  O/H 
MTBF 

2000  O/H 

MTBF 

Flying  Hours  per  year 

300 

300 

K  factor 

1.5 

1.5 

Operating  Hours  Per  Year  Per  Set 

450 

450 

Failures  Per  Year  Per  Set 

2.25 

.225 

Cost  of  Repair 

$500 

$500 

Depot  Level  Repair  Cost  Per  Set  Per 

Year 

1125 

112.50 

Total  Depot  Level  Repair  (5000  Sets) 

5,625,000 

$562,500 

Year  Cost  Savings  $5,062,500 
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FIGURE  F-12.  Depot  Level  Repair  Costs 


The  savings  in  depot  repair  costs  per  set  per  year  are  approxi¬ 
mately  6%  of  the  acquisition  cost  of  the  equipment.  Cost  savings 
for  other  repair  cost  assumptions  are  shown  in  Fig.  F-13,  based 
on  450  and  900  operating  hours  MTBF. 

It  should  be  recognized  that  this  savings  does  not  include  any 
reduction  in  field  level  maintenance  or  for  spares.  Including 
these  factors  could  substantially  increase  the  calculated  savings 
and  reduce  the  payback  period.  Additionally,  Gould  reported  that 
this  result  was  achieved  at  no  increase  in  unit  price  to  thf;  Navy. 
The  costs  incurred  to  screen  the  parts  were  offset  by  the  savings 
from  less  rework  during  manufacturing. 

4 — •  450  OP  HR  /  YR  » — *  900  OP  HR  /  YR 


COST  PER  REPAIR 


FIGURE  F-13.  ARN-84  Depot  Cost  Savings 
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5.  Part  Costs  Considerations  for  Weapon  Systems 


The  costs  of  parts  used  in  electronic  systems  represent  a 
substantial  part  of  the  total  costs  for  the  system.  Historically, 
standards  have  been  developed  to  ensure  that  the  desired  or 
required  quality  part  was  available  and  used  for  military  systems. 
MIL-HDBK-217D  describes  in  detail  the  quality  level  and  quality 
factors  associated  with  various  classes  of  parts.  Claims  are 
made  from  time  to  time  that  commercial  parts  could  be  used  in 
military  applications  to  achieve  the  same  results  if  they  were 
screened  properly.  While  this  thought  is  intuitively  appealing, 
we  have  no  data  to  support  the  claim.  In  fact,  based  on  failure 
rate  data  for  the  wide  variety  of  parts  found  in  MIL-HDBK-2 17D, 
quite  the  opposite  may  be  true. 

No  attempt  has  been  made  here  to  address  the  claim  that 
commercial  parts  can  do  the  same  job  as  MIL-STD  parts  if  screened 
properly  as  we  believe  this  is  a  separate  issue.  Our  analysis 
looks  at  the  total  cost  implications  of  parts  given  the  failure 
rates  that  have  been  observed  and  are  documented  in  MIL-HDBK-2 17D. 
Microelectronic  devices  were  selected  for  this  analysis  and  the 
failure  rates  associated  with  varying  part  quality  levels  are 
shown  in  Fig  F-14. 

Microelectronic  devices  were  selected  for  this  analysis 
for  several  reasons.  First,  the  radar  systems  case  studies 
document  that  35-40  percent  of  the  electronic  parts  are  micro- 
circuits.  Secondly,  the  devices  have  very  low  failure  rates  and 
it  was  felt  that  any  conclusions  reached  based  on  these  low 
failure  rate  devices  could  be  extrapolated  to  the  other  higher 
failure  rate  components  that  make  up  the  electronic  systems. 


Our  analysis  is  based  on  the  following  key  assumptions: 


(1)  Three  different  part  failure  rates  are  used.  These 
rates  are  extracted  from  MIL-HDBK-2 17 D.  MIL-M  38510  parts, 
quality  level  B  have  a  failure  rate  of  1.0.  Quality  level  B-l 
and  B-2  have  failure  rates  of  3.0  and  6.5,  respectively. 
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Procured  in  full  accordance  with  MIL-M-38510,  Class 


E 

RJ 

03 

* 

vo 

4J 

RJ 

■C 

C 

a  (Q 

0) 

cu 

■ — 

(0  o 

s 

o 

pH  -rO 

0 

— . 

V) 

If)  vo 

•u 

0) 

■H 

4-» 

c 

o  c 

3 

03 

0 

pH  (J) 

o’ 

•rH 

e 

in  o' 

id 

a) 

so 

u 

Ll 

,H  ’O 

03 

-«H 

1  -H 

OQ 

4J 

3 

£  pH 

O 

O’ 

1  RJ 

V) 

3 

0) 

J  > 

10)  T3 

Vo 

M 

1  03 

o 

£  T3 

in 

Ll  *-H 

C 

•o 

a. 

03 

id  rj 

1 

o 

rH 

1  * 

V 

•rH 

-O  4J 

lO 

0) 

U 

os  c 

li-H 

■rO 

4J 

o  a> 

• 

liD 

0-1 

O 

•rH  Ll 

Q 

X 

•H 

0 

rH  U 

m 

H 

rH 

a  3  « 

I 

03 

0 

a  o 

c 

S 

3 

03 

03 

l 

O'  ID 

jj 

J 

x: 

a>  au 

C 

■U 

JQ  <u 

X 

0 

oJ  O 

in 

rH 

X 

a 

c 

■a  p-h 

o-i  <d 

3 

4J 

a> 

<0 

o 

o 

•H 

4-> 

■a 

vo 

'» 

03 

o 

•  0 

O 

i 

•H 

0J 

to  3 

id 

rH 

•w  o 

vo 

o 

■o 

•H  pH 

0 

c 

•u 

d) 

E  H  '4J 

<0 

0 

oJ 

■H  <0 

-O 

c 

U) 

pH 

■o 

Jj 

<U 

ID 

ID 

o 

03 

oJ  -o  u 

OJ 

o 

•H 

C  RJ 

3 

o 

ID 

(0 

OJ 

(0 

id  jq 

in 

•H 

o 

in  u 

4-> 

H 

•*H 

rH 

C  ID 

03 

rH 

>  *H 

o  > 

•Q 

3 

id 

<0 

•H  *»H 

3 

«4H 

-o  £ 

oj  rj 

03 

in 

■H  3 

•c 

0J 

o 

E9 

•-H 

RJ 

ID 

C  0 

jq 

f 

£ 

o 

o  Z 

I'D 

OJ 

•iH 

O 

> 

> 

<0 

Vo 

0-1 

a> 

*  • 

E 

3 

O 

a 

a) 

•o 

03  4J 
U  0 

rj 

o 

o 

ID 

03  0 

OJ 

vo 

X 

jC 

4J  X 

<0 

Qo 

0) 

H 

0)  0) 

■o 

O 

00 

1 

00 

o 

• 

ro 

1 

03 

C 

T5 

E  <D 

Li 

■  H  • 

0 

RJ  O 

<T3  'V 

0  03 

C,  OJ 

U  C 

0) 

T3 

4J  ^ 

OJ  O 

R3  RS 

0)  4-3 

C 

c  c 

ID 

a  e 

V4  3 

03 

O  0 

£  If) 

1j 

03  4J 

o  E 

OJ 

pH  0 

>  -H 

03 

0 

•  C 

R3  0J 

■H  4J 

a* 

r-  H 

m  o 

O  C 

03  03 

c 

03  .H 

oo  g 

•H  0 

5  jQ 

•  H 

-  3 

00  ID 

U  O 

3 

3  Q  O’ 

1  u 

OJ 

O  03 

03 

U  ID 

Q  -H 

o  >z 

U 

in  u 

Eh  3 

03  4J 

D 

X3 

* — 

CO  O’  H  .pj 

JQ 

ID 

1  03 

03  rH 

• 

CJ 

If)  CJ 

*4  Lj 

(0 

35  >i  in 

O'  c 

M 

U  3 

03 

C  RJ 

X  r-i 

O  D1  03  6  Q 

•H  E 

03 

4J 

03 

3  kJ 

^h  a 

O  03 

03  03 

0 

RJ  O 

O  *H 

RJ  C.  pH  OJ 

4J 

Jj  Jj 

Ll 

U  4J 

U  RJ 

■o  C 

03  4-3 

4J 

T3 

0) 

O 

4J  C3 

C  rH 

ID  pH  O 

C  03 

O  -H 

m  a 

■iH 

O 

03  rH 

O  03 

o  -H 

rH 

U  >i 

E  <D  \ 

O  Li 

a<  oJ  oJ 

(D 

>j  o  m  id 

a  c  .-j 

Vo  J= 

0  oj 

c 

03 

0  pH 

•pH  OJ 

■a 

T3  0 

O  RJ 

3  -pH 

c  -o 

0  O'  >1 

3 

O’  3 

a)  D 

r. 

Lj 

C  O’ 

ID 

>  OJ 

OJ  T5 

O 

0 

Vo  ID 

in 

ID  -H 

Ol-H  TJ 

CJ 

U  ID  £  -H 

0 

OJ  C 

2S 

O  0J 

03 

4J 

RJ  R) 

•*  > 

03 

CJ 

•H  O 

*  03  m 

U 

-pJ  O' 

c  vo 

Ul  13  CO  3 

OJ  C 

ID  O 

O' 

X  C 

03 

•H  -pH 

ID  U 

C  pH 

1  03 

•H 

CJ  c 

Vo  O 

•  H  rH  Q 

x: 

ID  ID 

O  <0 

3  RJ  Eh  OJ  Eh 

CU  ID 

in 

RJ  £ 

C/5  C 

in  u 

c 

Ij  10 

1  V 

CJ 

0)  -H 

•o 

J  U 

• 

•o  If) 

-C 

D 

H  Jj 

Q 

D 

OJ  XI 

U  O 

£  3 

Vo  jJ 

c 

in  -.j 

U 

TJ 

RJ  1 

pH  in 

M  >  0J 

C 

am 

pH 

a  03 

0  OJ 

RJ 

4) 

RJ  OQ 

'V 

a 

Jj  ID 

in  ai 

U 

a  > 

0  in 

O  03 

oJ  o 

0 

oj  in 

rH  £ 

c  X 

in 

Vo  JQ 

R! 

in  H 

ID  ID 

a  o  rj 

X3  HO 

X 

E 

3 

.U 

<D  CJ  ro 

ID  T3 

0 

o  0 

ij 

i  • 

U  ID 

Li 

03  SZ 

3  • 

£  in 

•H  3 

O 

U  4J 

o  -r 

1  Jo 

3  0 

4J 

0  o 

pj  ai 

ct-h 

L4 

C  Ol 

u  o 

H  OJ 

ID  pH 

O 

0  c 

Op  m 

£  D 

Jj  RJ 

iu 

O  -H 

rH 

1 

CD 

o-i  aj  i  a 

o  >  c  a  at 
o  <  c 
w  a>  o  -m 


0 

4-J 

rH  0) 

o 

Li 

•H 

03  03 

0) 

■rH 

3 

3  03 

3 

O' H  IN 

O’  D 

u 

1 

0 

o 

4J 

CD 

Li 

c 

c  - 

03 

0  m 

ID 

O'  •O 

rH  o 

> 

c 

Li 

03  o 

O 

‘H 

O 

>  in  jq 

c 

a 

♦rH 

03 

ID 

o 

3  'O 

ID 

03 

C1  0 

0 

Vo 

0  -C 

-C 

O 

c 

4J 

4J 

in 

•rH 

03  0 

-  X 

03 

3) 

-a 

u 

c 

JQ 

c 

0  * 

•rH 

OJ 

0 

-o  x 

c 

C  X 

*iH 

100 

* 

0  X 

03 

O 

lid 

>  1 

4J 

3  'Jl  in  rj  o  O’ 
o 


i  o 

a  o-i  ^ 
H  O  CJ 
c/J 

I  If)  in 

pj  OJ  a 


£  a)  o 

E  lo 
u  id  U 


|tn  3  o 

loJ  O'  o-i 


(2)  Microcircuit  costs  range  from  $1  to  $130.  On  average, 
costs  per  part  are  in  the  range  of  $6-7. 

(3)  As  a  class,  microcircuits  have  a  failure  rate  of  1.5 
per  million  part  operating  hours. 

(4)  MIL-M  38510  parts  cost  twice  as  much  as  vendor  equiva¬ 
lent  parts  screened  to  MIL-STD  883  requirements. 

(5)  The  cost  to  repair  is  dependent  on  how  soon  the  failure 
is  identified.  These  costs  are  shown  in  Fig  F-15. 

(6)  Systems  contain  5000  microcircuits. 


LEVEL 


PART 

1.00 

SUBASSEMBLY 

75.00 

ASSEMBLY 

200.00 

SYSTEM 

1000.00 

FIELD 

2000.00+ 

FIGURE  F-15.  Repair  Cost  Per  Failure 

A  number  of  excursions  were  made  to  determine  the  part  pre¬ 
ference  break-even  point.  Details  of  those  excursions  are  contained 
in  Appendix  C.  Figure  F-16  shows  the  resultant  part  preference 
based  on  an  average  cost  per  part  of  $7.  Even  though  higher 
average  part  costs  tend  to  move  the  curve  to  the  right,  the 
overall  conclusions  do  not  change  until  the  average  part  cost 
reaches  approximately  $100. 
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SUB  t  PRIME  CONTRACTOR 
PRE-OELIVERY  TESTING 
(approx  500  hrs) 


PREFERRED 


7  AVERAGE  PART  COST 
A I  LURE  RATE  FACTOR  3  S  6 


MILITARY  ISlPREFERRED 


MILITARY 
SERVICE 
USAGE 
(5000  HRS# 
10  YRS) 


AVERAGE  REPAIR  COST  (t> 

FIGURE  F-16.  Microcircuit  Resultant  Part  Preference 

Figure  F-16  depicts  two  bands  of  data  that  reflect  slightly 
different  conclusions.  The  bands  are  a  result  of  plotting  the 
total  costs  of  MIL-M  38510  parts  divided  by  the  total  costs  of 
vendor  equivalent  parts  (commercial)  at  various  average  repair 
costs  for  the  two  failure  rates  listed.  The  upper  curve  of  each 
band  is  for  a  failure  rate  three  times  higher  than  demonstrated 
for  the  MIL-M  38510  part  and  the  lower  curve  is  for  6.5  higher. 
The  center  (dotted)  line  represents  the  break-even  point  for  each 
combination  considered. 

One  additional  factor  has  been  used  to  complete  the  analysis 
as  shown.  Notice  that  the  upper  band  is  based  on  an  equipment 
usage  of  500  hours  and  the  bottom  band  5000.  These  operating 
hours  were  selected  to  represent  the  number  of  part  operating 
hours  that  a  typical  equipment  might  see  in  the  factory  during 


the  manufacturing  process  and  subsequently  in  field  use.  For 
example,  a  system  with  5000  microcircuits  would  expect  to  see 
approximately  four  part  failures  during  the  first  500  hours  of 
operation  when  MIL-M  38510  microcircuits  were  used.  During  the 
same  period,  a  vendor  equivalent  part,  quality  level  B-2  would 
expect  to  see  approximately  25  failures  during  the  same  period. 

Based  on  this  analysis,  parts  with  low  failure  rates  as 
demonstrated  by  the  MIL-M  38510  parts  are  preferred  to  parts 
that  have  exhibited  higher  failure  rates  when  the  costs  of  repair 
is  high  even  though  the  operating  hours  might  be  low.  When 
operating  hours  are  high,  as  might  be  expected  with  most  equipment 
delivered  to  the  military,  the  lower  failure  rate  part  is  preferred 
at  virtually  all  expected  repair  cost  levels.  Figure  F-17  depicts 
these  part  preference  choices. 


FIGURE  F-17.  Part  Preference  Choices 
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This  analysis  does  not  answer  all  the  questions  one  might 
ask  concerning  the  cost  and  benefits  of  military  versus  commercial 
parts.  It  does,  however,  point  out  that  if  commercial  parts  can 
do  the  job  required  of  the  equivalent  military  part,  they  must  be 
prepared  to  demonstrate  failure  rates  that  approach  those  of  the 
MIL-M  38510  parts  if  they  are  to  be  cost-effective  from  the 
military  point  of  view  even  though  their  acquisition  costs  might 
be  considerably  less. 
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The  High  Cost  of  CNDs 


Throughout  the  case  studies,  problems  caused  by  inadequate 
diagnostics  are  legion.  The  diagnostics  problem  is  discussed 
in  the  main  report  and  in  Appendix  E.  This  section  quantifies 
the  costs  associated  with  CNDs,  (cannot  duplicate  or  no  defect 
maintenance  events)  and  provides  the  incentive  to  accelerate 
diagnostics  improvements. 

For  example,  during  the  period  1980-1982,  approximately 
70  percent  of  all  corrective  maintenance  events  on  the  F-15  and 
F-16  radar  systems  were  for  CND  actions.  During  the  same 
period,  between  30  and  50  percent  of  all  corrective  maintenance 
man-hours  were  consumed  in  response  to  CND's.  These  percentages 
are  shown  in  Fig.  F-18  for  the  APG-63  and  in  Fig.  F-19  for  the 


APG-66 . 


Another  important  measure  of  the  effect  of  CNDs  on  the 
maintenance  function  can  be  seen  in  Figs.  F-20  and  Fig.  F-21 
for  the  APG-63  and  APG-66,  respectively.  For  the  period,  the 
rate  of  CND  events  per  flying  hour  for  the  radar  alone  averaged 
.10  for  the  APG-63  and  .07  for  the  APG-66.  Rate  trends  are 
apparent  in  the  figures  as  presented. 

The  actual  costs  attributable  to  each  CND  have  been  calcu¬ 
lated  and  are  presented  in  Fig.  F-22.  Costs  were  calculated 
based  on  data  from  the  AFLC  K051,  D056B  and  D056T  analysis 
products.  The  average  cost  per  MMH  is  based  on  a  known  1980 
labor  rate  of  $35  per  maintenance  man-hour  for  logistics 
support  cost  computation  contained  in  the  K051  report.  The 
average  cost  of  $30.87  and  $12.20  per  CND  per  flight  hour  for 
the  APG-63  and  the  APG-66  are  conservative,  given  the  inaccurac¬ 
ies  of  the  reporting  systems,  but  still  point  out  the  magnitude 
of  costs  that  can  be  avoided  by  improving  the  diagnostic  capa¬ 
bilities  of  the  radar  systems. 
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FIGURE  F-I8.  APG-63  Maintenance  Events  Summary 
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TEST  ASSETS 


The  determination  of  the  need  and  commitment  of  assets  required 
for  a  properly  executed  reliability  and  maintainability  program 
is  critical  to  the  success  of  a  program.  Due  to  budget  restraints 
there  is  a  tendency  to  minimize  the  expenditure  for  test  assets 
in  almost  all  development  programs.  This  results  in  a  high  likeli¬ 
hood  of  there  being  too  few  assets  to  accomplish  program  critical 
testing.  A  scarcity  of  test  assets  often  causes  management  to 
place  a  lower  level  of  priority  on  reliability  and  maintainability 
testing  than  on  other  test  activities.  For  these  reasons  it  is 
important  that  adequate  assets  be  planned  for  R&M  activities.  If 
reallocations  or  reassignment  of  the  R&M  test  assets  are  considered, 
the  impact  on  R&M  should  be  assessed. 

In  a  well-planned  integrated  test  program,  R&M  data  should 
be  obtained  from  the  performance  evaluation  and  environmental 
qualification  tests  to  be  used  as  soon  as  possible  in  an  R&M 
growth  test  program. 

The  economics  of  effective  R&M  development  are  dependent  on 
early  testing  and  early  corrective  action.  The  cost  for  correc¬ 
tive  action  increases  rapidly  as  program  time  passes  and  it  may 
not  be  cost-effective  to  incorporate  changes  as  production  advances. 

1 .  Assets  for  Reliability  Testing 

Assets  are  required  for  reliability  development  testing  and 
reliability  qualification  tests,  to  assure  that  the  design  activ¬ 
ities  have  in  fact  addressed  the  stress  and  environments  that 
are  expected  in  operation. 

a .  Reliability  Development  Test 

Reliability  development  tests  are  needed  to  identify  failure 
modes  not  detected  by  analysis.  Development  testing  is  intended 
to  find  failure  modes  and  to  provide  information  on  ways  to  develop 
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a  corrective  action  for  these  failure  modes.  The  assets  for  the 
reliability  development  tests  need  to  be  available  early  in  the 
development  cycle,  and  should  therefore  be  allocated  the  first 
units  available.  This  allows  the  gathering  of  early  failure  data 
and  incorporation  of  corrective  actions  prior  to  or  near  the  begin¬ 
ning  of  production.  Since  the  configuration  of  the  early  units 
is  not  fully  representative  of  the  final  design,  reliability 
data  and  test  hours  should  be  attained  on  test  articles  which  are 
dedicated  to  other  activities.  The  F/A-18  APG-65  development  test 
program  used  the  approach  of  combining  hours  from  early  configura¬ 
tion  units  and  added  later  configurations  as  the  test  progressed. 

From  the  standpoint  of  gathering  hours  or  experience  on  various 
test  units,  the  real  measure  of  test  assets  needed  for  reliability 
development  tests  is  in  terms  of  test  hours  and  calendar  time  re¬ 
quired  to  obtain  these  test  hours,  because  of  the  impact  and  time 
sensitivity  of  the  information  that  these  tests  produce.  Effi¬ 
ciency  is  gained  if  multiple  units  are  used  at  any  one  time  rather 
than  a  single  test  unit.  The  minimum  requirement  then  is  for 
two  test  assets  to  be  assigned  by  specific  serial  number  to  relia¬ 
bility  development  test  and  change  them  as  later  configurations 
become  available. 

b.  Demonstration  Tests 

The  tests  provide  a  means  of  determining  contract  compliance 
and  provide  essential  engineering  data.  The  orientation  and  focus 
should  be  directed  toward  action  that  results  in  improvement  in 
the  reliability  of  the  system  as  well  as  demonstration  of  compli¬ 
ance  with  specific  reliability  levels. 

The  general  inclination  is  to  pursue  the  allocation  of  test 
assets  in  first  priority  for  demonstration  testing.  Demonstration 
testing  has  the  potential  to  field  valuable  development  data  if 
the  priority  of  test  success  is  not  pushed  to  exclusion  of  other 
data.  Allocation  of  assets  for  both  reliability  development  and 


demonstration  testing  is  in  fact  essential  for  a  successful  program. 
To  provide  an  adequate  statistical  basis  it  is  important  that  the 
demonstration  be  conducted  on  no  less  than  two  test  units. 

Test  articles  should  be  dedicated  for  use  during  the  reliabil¬ 
ity  demonstration  test  time  period  and  may  have  other  applications 
after  the  test.  Reliability  development  and  demonstration  tests 
should  be  conducted  as  early  in  FSD  as  hardware  schedules  allow, 
with  development  tests  conducted  prior  to  demonstration  tests. 


Since  changes  can  occur  within  the  production  process,  to 
assure  that  the  quality  of  the  hardware  is  maintained  throughout 
the  production  cycle,  reliability  demonstration  tests  on  subsequent 
production  systems  should  be  considered.  Periodic  production  re¬ 
liability  tests  have  been  conducted  on  the  F-15  and  F-16  radars 
and  have  identified  problems  that  were  corrected. 


2.  Assets  for  Maintainability  Testinc 


Test  assets  are  needed  for  a  program  to  develop  and  mature 
the  BIT  system.  In  addition,  principal  needs  include  the  vali¬ 
dation  of  built-in  test  (BIT)  and  maintainability  demonstrations, 


a .  BIT  Development/Maturation 

Fault  insertion  approaches  have  been  shown  to  be  of  value 
if  the  fault  lists  are  realistic  and  comprehensive.  The  assign¬ 
ment  of  a  single  test  asset  may  be  adequate  if  the  asset  is 
representative  of  the  BIT  design  and  has  the  supporting  software 
of  the  appropriate  configuration.  Experience  has  indicated  that 
it  is  almost  impossible  to  design  an  effective  BIT  system  from 
theoretical  data  or  speculation  of  fault  patterns.  A  good  BIT 
system  must  be  developed  and  matured  by  the  individual  pursuit 
of  fault  experience  and  diagnostic  process.  For  this  necessary 
process  to  take  place  it  is  absolutely  necessary  to  provide  a 
test  asset  system  near  a  location  where  failures  are  generated — 
this  test  system  being  available  to  those  who  are  continuing  to 
grow  and  improve  the  BIT  system. 
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The  basis  for  BIT  maturation  is  properly  the  number  of  faults 
that  are  experienced  rather  than  the  number  of  test  operating  hours 
or  the  number  of  calendar  days.  The  efficiency  of  the  maturation 
program  depends  upon  the  realism  of  the  fault  list  and  the  effi¬ 
ciency  of  the  engineering  actions  that  are  taken  as  a  result  of 
the  information  that  is  obtained.  Data  should  also  be  obtained 
from  the  other  tests  and  used  in  maturing  the  BIT  system. 


APPENDIX  G 


INFORMATION  SYSTEM  OBSERVATIONS 


Throughout  the  course  of  this  study,  the  number  of  times 
that  the  requirements  for  "good  data"  presented  itself  is 
legion.  Beginning  with  earliest  design  activity  and  continuing 
throughout  the  service  life  of  the  equipment,  good  decisions  can 
be  seen  to  stem  from  good  data  inputs.  In  this  appendix,  we 
examine  several  events  where  "official  data"  were  used  to  tell  a 
story  or  depict  an  event  when,  in  fact,  a  different  story  might 
have  been  apparent  if  a  more  in  depth  effort  was  made  to  under¬ 
stand  what  the  data  were  reporting.  The  overall  message  from 
this  portrayal  is  to  suggest  to  the  reader  that  data  must  always 
be  questioned  for  hidden  meanings.  In  addition,  we  point  out 
existing  features  of  service  data  collection  systems  and  evaluate 
those  systems  in  light  of  future  requirements. 

Wide  variations  in  results,  using  data  obtained  from 
the  same  data  base,  have  been  observed  in  the  course  of  this 
study.  Six  examples  are  provided  as  follows: 

A.  VARIATIONS  IN  THE  USE  OF  DATA 

1.  Impact  of  the  F-16  Fleet  Electrical  Modification 

In  August  of  1981,  the  F-16  fleet  was  down  for  a  modification 
of  the  electrical  system.  The  number  of  flying  hours  for  August 
was  25  percent  of  that  flown  in  July  (1,344  versus  5,835)  and 
recovered  to  only  81  percent  in  September  (4,734  versus  5,835). 

The  impact  of  the  modification  is  clearly  displayed  in  Fig.  G-l, 
where  all  three  measures  show  large  dips.  Figure  G-2  contains 
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five  bar  graphs  of  the  1981  data,  bracketed  on  each  side  by  the 
prior  year  (1980)  and  the  following  year  (1982).  The  first  1981 
bar  presents  the  entire  year's  data,  the  second  1981  bar  removes 
the  third  quarter's  data,  the  third  1981  bar  is  the  third  quarter 
by  itself,  the  fourth  1981  bar  removes  the  August  data  and  the 
fifth  1981  bar  is  the  fourth  quarter  data  only.  Figure  G-3  shows 
the  impact  of  the  fleet  electrical  modification  on  reliability 
growth  plots.  Note  that  once  the  August  data  are  removed,  the 
growth  line  is  nearly  straight. 

2.  F-15  and  F-16  Radar  Removal  Data 


Many  avionics  experts  depend  on  LRU  removal  rates  as  the 
key  measure  of  system  "goodness."  The  supporting  rationale  for 
use  of  the  removal  rates  is  quite  convincing,  i.e.,  removal 
rates  (less  removals  for  access  and  cannibalizations)  are  much 
more  indicative  of  the  total  health  of  the  system.  They  also 
include  the  impact  of  diagnostic  problems,  which  in  turn  affect 
skill  and  training  requirements.  Inherent  reliability,  on  the 
other  hand,  is  not  indicative  of  the  total  maintenance  burden 
and,  in  fact,  is  not  intended  to  do  so  as  it  does  not  include 
the  impact  of  faulty  diagnostics,  etc.  Total  maintenance  actions 
as  a  measure,  is  much  closer  to  removal  rates  in  its  ability  to 
measure  the  total  maintenance  burden  as  it  includes  the  impact 
of  diagnostics,  cannibalization,  remove  for  access,  minor  mainte¬ 
nance  actions,  etc.  The  reason  that  removal  rates  are  generally 
used  as  a  key  maintenance  indicator  is  that  they  drive  most  of 
the  other  levels  of  maintenance,  e.g.,  intermediate  and  depot 
levels,  and  thus  the  associated  resources  at  those  levels,  e.g., 
support  equipment,  transportation,  manpower,  etc. 
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Accepting  the  premise  that  removal  rate  is  the  single  most 
important  measure,  the  next  question  is  where  does  one  get  the 
data,  especially  since  accuracy  is  very  important.  A  review 
of  studies  of  the  accuracy  of  the  Service  data  systems  indicate 
that  as  few  as  2  percent  of  the  maintenance  records  are  totally 
accurate  (every  single  entry  is  precisely  correct)  and  that  40 
percent  to  70  percent  of  the  records  are  "functionally  useful" 
(accurate  enough  to  track  trends).  Most  analysts,  therefore, 
draw  their  removal  data  from  the  supply  system  or  an  off-line, 
manual  system.  Some  of  the  pitfalls  of  obtaining  accurate  removal 
data  are  illustrated  on  Figs.  G-4,  G-5,  and  G-6.  Figure  G-4 
presents  data  obtained  from  a  Rand  Corporation  briefing  as  well 
as  data  from  the  Air  Force  D056  data  system.  The  Rand  data  were 
derived  from  the  AFTO  Form  95  log  in  the  Avionics  Intermediate 
Shop  ( AIS )  at  each  base.  The  D056  data  are  for  all  F-15/F-16 
bases  for  the  same  period  of  time.  Note  that  the  F-15  C/D  D056B 
data  adjusted  to  the  same  basis  (per  sortie  versus  per  flying 
hour)  is  about  30  percent  higher  than  the  Rand  data,  while  in 
both  F— 16  comparisons  the  D056B  data  are  more  than  50  percent 
lower.  Figure  G-5  also  uses  D056B  and  is  the  same  subset  of 
removals  (i.e.,  without  cannibalizations  and  remove  from  access) 
as  Fig.  G-4.  These  data  show  that  the  F-16  radar  removal  rate 
(on  average)  is  about  three  times  better  than  the  next  best 
(F-15  C/D).  The  chart  also  shows  that  the  F-16  rate  is  extremely 
variable  and  consistently  so.  Figure  G-6  is  also  data  from  a 
Rand  briefing  on  the  variability  of  demand  rates  on  specific 
parts  (indicated  by  the  five-digit  work  unit  code)  both  at  a 
base  as  well  as  between  bases.  The  message  is  that  there  are 
large  variations  at  the  same  base  over  time  as  well  as  between 
bases.  Field  interviews  at  the  bases  were  used  to  explain  the 
variation. 
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in  most  cases  reasonable  explanations  were  found.  The  reasons 
varied  from  organizational  changes  to  specific  modifications  to 
parts  and  automatic  test  equipment. 

Given  the  above  situation,  even  if  the  AFTO  Form  95  derived 
data  are  more  accurate,  the  manpower  costs  of  capturing  the 
data,  which  results  in  3  to  6  months  of  data  for  a  few  bases, 
could  be  very  misleading.  The  other  two  approaches  have 
limitations  as  well.  The  D056B  data  are  suspect  as  to  accuracy, 
but  easily  available,  while  the  base  level  supply  data  are  likely 
more  accurate,  but  require  special  data  runs  and  must  be  captured 
on  a  base-by-base  basis. 

3 .  Impact  of  Installation  of  Maintenance  Data  Terminals 
at  All  F-16  Bases 

Beginning  in  mid-1982,  maintenance  data  terminals  were 
installed  at  all  F-16  bases.  By  late  1982  or  early  1983,  the 
installation  was  complete.  During  that  period,  based  on  infor¬ 
mation  from  the  F-16  SPO,  over  3,500  job  control  numbers  that 
were  in  limbo  in  the  data  system  were  closed  out  in  order  to 
mechanize  the  data  process.  The  regression  of  the  radar  MTBM(T), 
(see  Fig.  G-7),  the  middle  line,  from  the  low  twenties  to  about 
17  is  likely  a  reflection  of  that  conversion.  The  leveling  of 
inherent  reliability  could  also  be  affected  by  the  conversion. 

The  following  section  reflects  a  historical  precedent  for  this 
supposition. 

4.  Impact  of  Installation  of  Data  Terminals  at  Dover  AFB 
for  the  C-5A 

Figures  G-8  and  G-9  show  the  impact  of  automating  the  main 
tenance  data  process  at  Dover  AFB,  Delaware  (C-5A  data).  At 
Dover,  installation  of  terminals  and  the  subsequer*-  in-line 
editing  and  checking  of  maintenance  tasks  have  resulted  in  much 
822/3-10 


39Yd3AY 


33dHi 


so* 


more  of  the  maintenance  tasks  are 


more  accurate  data,  i.e., 
captured  in  the  data  system  and  the  entries  are  much  more  accurate 
The  Automatic  Maintenance  System  (AMS)  at  Dover  is  much  further 
developed  than  the  F-16  system  and  in  fact  after  an  initial  down¬ 
turn  of  reliability  and  maintainability  measures  at  Dover  AFB , 
due  again  to  capturing  of  more  and  more  accurate  data,  the  R&M 
measures  now  show  significant  improvement,  i.e,  27  percent  less 
parts  used  per  sortie,  33  percent  increase  in  MTBM(T),  89  percent 
decrease  in  abort  rate,  and  41  percent  increase  in  maintenance 
productivity.  These  improvements  are  not  just  data  improvements, 
but  reflect  the  potential  for  significant  maintenance  improvements 
via  improved  planning,  scheduling  and  controlling,  as  well  as 
providing  near-real-time  data  to  the  maintenance  man,  allowing 
him  to  work  smarter.  Similar  results  may  occur  on  the  F-16. 

5.  T-38  Base-to-Base  Comparison  of  Avionics  Reliability 

A  recent  study  of  operational  influences  on  avionics  relia¬ 
bility  examined,  among  many  other  variables,  how  the  field  opera¬ 
tional  MTBF  of  three  different  avionics  equipments  on  the  USAF 
T-38  twin  jet  trainer  aircraft  compared  across  nine  different 
Air  Training  Command  bases  flying  essentially  the  same  basic 
pilot  training  program  over  a  period  of  one  calendar  year.  The 
results  (Fig.  G-10)  indicated  that  the  equipment's  MTBF  performance 
(expressed  as  a  percentage  of  the  average  performance  at  all 
nine  bases)  varied  from  as  low  as  57  percent  at  the  base  with 
the  lowest  MTBF  to  a  high  of  285  percent  at  the  base  with  the 
highest  operational  MTBF.  Similarly,  when  the  combined  results 
of  the  composite  performances  of  all  three  equipments  (UHF  Radio, 
TACAN,  Inertial  Platform)  were  compared,  similar  differences  of 
78  percent  and  163  percent  were  observed.  Upon  more  detailed 
investigation,  the  conclusions  drawn  were  that  the  differences 
observed  were  due  primarily  to  differences  in  the  maintenance 
skill  levels  across  the  nine  different  bases. 
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For  example,  one  might  conclude  that  the  rapid  and  sustained 
growth  for  the  APG-66  radar  reliability  was  due  to  the  fact  that 
field  usage  began  very  soon  after  the  first  production  delivery. 
This  very  short  time  from  production  to  field  use,  coupled  with 
high  production  rates,  enabled  the  contractor  to  learn  of  field 
problems  early  and  to  correct  them,  which  led  to  large  improvements 
in  reliability. 

Another  point  that  can  be  made  is  that  the  Navy's  "New  Look" 
program  is  probably  working  since  initial  field  values  for  APG-65 
reliability  are  considered  quite  good.  A  part  of  the  "New  Look" 
program  calls  for  early  reliability  development  testing.  As  can 
be  seen  in  Figure  G-12,  the  APG-65  did  have  a  significant  number 
of  growth  test  hours  during  FSD.  That  could  have  contributed 
to  this  high  initial  field  reliability,  even  though  others  have 
pointed  out  that  very  little  new  was  learned  during  the  reliability 
growth  test  phase. 

Still  another  point  that  looks  obvious  is  the  visual  pre¬ 
sentation  of  the  APG-63  C/D  as  compared  to  the  APG-63  A/B.  One 
could  conclude  that  as  the  C/D  received  the  attention,  the  A/B 
reliability  declined,  or  that  the  C/D  represents  normal  growth  of 
an  extended  A/B  radar  program. 

All  of  these  observations  are  made  based  on  data  plotted  on  a 
linear  scale,  even  though  we  all  know  that  the  scale  should  be  log¬ 
arithmic  when  this  type  of  comparison  is  made.  The  conclusions 
reached  from  this  chart,  however,  may  be  valid  for  the  intended 
purpose,  but  as  with  all  the  other  data  examples  that  precede  this 
one,  more  information  is  probably  needed  before  any  judgment  is 
made . 
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B.  USE  OF  SERVICE  DATA  SYSTEMS 

1.  General 


During  the  course  of  the  study  significant  differences  in  R&M 
performance  were  noted,  depending  on  how  the  data  were  sliced  and 
how  well  other  confounding  factors  were  discovered,  investigated 
and  understood.  The  utility  of  the  data  depends  on  the  question 
to  be  answered.  As  long  as  trending  data  can  be  used  to  answer 
the  question,  the  Service  data  systems,  if  used  cautiously  and 
properly,  can  provide  useful  information.  The  six  examples 
previously  discussed  demonstrate  some  of  the  pitfalls:  impact 
of  events  such  as  the  F-16  Fleet  Electrical  Modification;  potential 
inaccuracies  due  to  small,  limited  sample  sizes  as  illustrated  in 
the  F-16  and  F-15  radar  removal  rate  variation,  both  between  and 
within  operational  locations;  significant  changes  in  basic  R&M 
measures  due  to  changes  in  data  collection/reporting  procedures 
such  as  at  Dover  AFB,  Delaware  and  the  F-16  data  terminals;  the 
impact  of  other  support  elements  such  as  training  and  skill  levels 
on  the  (apparent)  reliability  (operational  MTBF)  of  equipment 
illustrated  in  the  T-38  avionics  example.  The  utility  of  the  data 
for  making  engineering  decisions  like  "what  to  fix,"  when  all 
indicators  say  the  reliability  is  too  low,  is  suspect.  As  illus¬ 
trated  above,  the  real  problem  could  be  unrelated  to  the  reliability 
of  the  equipment.  It  could  be  a  reflection  of  poor  training;  a 
requirement  for  skills  beyond  the  level  of  typical  maintenance 
personnel,  poor  technical  orders;  the  use  of  the  equipment  outside 
of  its  designed  limitations,  inadequate  support  equipment,  poor 
fault  detection  and/or  fault  isolation  capability,  etc.  Acquiring 
data  which  have  the  attributes  required  to  analyze  and  pinpoint 
the  cause  of  such  problems  presents  a  dilemma.  The  principal 
intent  of  the  systems  is  to  determine  resource  requirements  and  to 
identify  problems.  The  question  is,  to  what  degree  should  the 
data  systems  provide  the  information  to  support  analysis  of 
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resource  requirements  and  to  determine  specifically  what  needs 
to  be  done  to  fix  problems.  The  data  analysis  problems  discussed 
above  provide  some  insight  into  the  lack  of  current  capability. 

The  shortfall,  with  the  exception  of  the  diagnostics  area,  is 
not  one  of  more  needed  data  elements — it  is  instead  missing  data 
and  inaccuracies  of  reported  data.  Diagnostics  data  need  improve¬ 
ments  across  the  board. 

2.  Solutions  to  the  Data  Dilemma 

Two  solutions  to  the  data  dilemma  have  been  examined.  First, 
the  Army  approach,  i.e.,  the  sample  data  collection  system  (SDC), 
which  includes  collection  of  logistics  data  via  a  sampling  system 
as  well  as  a  very  detailed  data  collection  during  the  acquisition 
process  via  the  RAMLOG  Data  System,  and  second,  the  Air  Force 
Automated  Maintenance  System  (AMS)  for  the  C-5A  aircraft  at  Dover 
AFB,  Delaware. 

The  Army  SDC  system  evolved  from  cognizance  of  the  same  problems 
described  in  the  limitations  above.  In  the  Army  case,  they  were 
faced  with  another  compounding  dimension,  namely,  dispersal  of  the 
maintenance  process  (e.g.,  five  levels  of  maintenance  versus  three 
for  the  Air  Force).  Their  solution  was  to  use  "dedicated"  data 
collectors  to  collect  accurate  logistics  data  via  a  sampling 
approach.  Since  data  are  their  business,  they  tend  to  be  highly 
motivated  to  ensure  that  the  data  are  accurate  and  timely.  In 
addition,  the  data  collection  effort  can  be  adjusted  in  frequency, 
number  of  locations  sampled,  and  degree  of  detail.  The  RAMLOG 
system,  developed  during  the  Blackhawk  helicopter  program,  has  also 
proven  to  be  an  excellent  data  system  for  the  acquisition  process. 

The  Army  SDC  system  is  proving  to  be  an  excellent  approach  for 
determining  resource  requirements,  identifying  problems  areas  and 
providing  detailed  accurate  data  required  to  support  analysis. 

The  Air  Force  AMS  approach  is  at  the  other  end  of  the  spectrum 
of  solutions  to  the  data  dilemma.  AMS  is  a  computer-based  real¬ 
time  system  which  uses  a  central  data  base  concept.  The  central 
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data  base,  located  at  the  Oklahoma  City  Air  Logistics  Center, 
(OC-ALC)  is  connected  to  on-line  terminals  at  Dover  AFB,  Delaware 
(over  100  terminals  in  maintenance  shops,  job  control,  wing  head¬ 
quarters,  etc.)  as  well  as  OC-ALC  and  San  Antonio  ALC.  The  system 
is  currently  being  extended  to  other  C-5A  bases  at  Altus  AFB,  OK 
and  Travis  AFB,  CA.  AMS  adds  at  least  two  dimensions  to  the 
utility  of  data  systems.  In  addition  to  providing  superb  data 
for  resource  requirements,  problem  identification,  and  data  to 
support  analysis,  the  system  provides  real-time  data  for  planning, 
scheduling  and  controlling  of  the  maintenance  process,  e.g., 
jobs  are  not  scheduled  until  all  of  the  necessary  resources 
(people,  parts  and  material,  support  equipment),  as  well  as  the 
end  item  to  be  fixed  (aircraft  or  line  replaceable  unit  (LRU)  are 
available.  The  benefits  accrued  from  this  added  dimension  are 
illustrated  in  Fig.  G-13;  home  station  logistics  caused  delayed 
departures  and  Fig.  G-14,  increase  in  fully  mission  capable 
rate,  while  they  may  appear  to  be  small  have  had  the  effect  of 
providing  two  to  three  more  aircraft  available  for  sortie  gener¬ 
ation.  The  second  added  dimension,  provided  by  AMS,  is  real-time 
availability  of  detailed  accurate  data  for  all  levels,  not  only 
for  analysis  but  also  for  the  day-to-day  maintenance  process. 

For  example,  prior  to  departing  the  shop  for  a  task,  maintenance 
personnel  are  provided  a  printed  history  of  the  system/subsystem 
on  the  aircraft  that  they  are  going  to  work  on.  The  effects  of 
this  powerful  capability  are  dramatic,  e.g.,  reduction  of  about 
20  percent  in  spare  parts  used  per  sortie  flown,  a  significant 
reduction  in  repeat  malfunctions  and  even  more  critical,  the  data 
now  provide  a  readily  available  tool  which  is  very  useful  to  the 
people  who  report  it  (the  basic  truth  of  good  data  systems — the 
data  are  useful  to  those  who  report  it).  AMS  is  a  powerful, 
proven  tool  that  has  provided  enormous  benefits  for  the  C-5As  at 
Dover.  However,  unlike  the  Army  SDC  system,  which  is  available 


822/3-21 


Army-wide  AMS  is  £ully  operational  only  at  Dover  AFB  (soon  to 
be  operational  at  the  remaining  C-5A  bases). 

Both  SDC  and  AMS  provide  useful  solutions  to  the  data  dilemma. 
AMS  provides  added  dimensions  and  should  be  considered  as  a  model 
for  future  data  systems.  SDC  provides  good,  accurate  data,  but  if 
the  data  are  not  being  collected  as  a  normal  part  of  the  sampling 
process,  time  is  required  to  request,  get  approval,  collect, 
process,  and  analyze  the  data. 

While  the  Army  SDC  provides  the  best  Service-wide  approach 
today,  it  would  appear,  based  on  Army,  Navy,  and  Air  Force  projected 
logistics  requirements  and  Year-2000  concepts,  that  data  systems 
modeled  around  the  AMS  approach  should  be  sought  as  the  data 
systems  of  the  future. 
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BACK-UP  DATA 


This  appendix  documents  data  from  official  Air  Force  and 
Navy  sources  that  was  used  for  various  analysis  throughout  the 
study  period.  The  Air  Force  data  is  from  the  D056B  and  D056T 
reports  and  the  Navy  data  is  from  3M. 
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FIGURE  H-19.  F4E  Radar  Maintenance  Summary 
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